不同服务器端模型在并发请求下的表现差异

一般来说,服务器的模型有下面几种:

  • 单进程单线程
  • 多进程(一个请求对应一个进程,Apache 之类貌似就是这个)
  • 多线程(一个请求对应一个线程,这个挺少见到的)
  • select / poll
  • epoll

它们的差别可以看这个,http://www.cnblogs.com/sharra/…

因为需要了解底层设备访问的原理,所以惯用高层应用语言的我,需要了解一下Linux的设备访问机制,尤其是处理一组非阻塞IO的原理方法,标准的术语好像是叫多路复用。以下文章部分句子有引用之处,恕没有一一指出出处。

对于接触过Linux内核或设备驱动开发的读者,一定清楚poll和select系统调用,以及从2.5版本引入的epoll机制(epoll机制包含三个系统调用)。网上关于它们的文章,有说用法的,甚为详细,更有分析源代码的,又比较深入,且枝节颇多。经过几篇文章的阅读,我把觉得比较核心的东西写下来吧。我的用意是尽可能以简单的概念,比对他们三者的异同。

几经查找我才确定下来,poll和select应该被归类为这样的系统调用,它们可以阻塞地同时探测一组支持非阻塞的IO设备,是否有事件发生(如可读,可写,有高优先级的错误输出,出现错误等等),直至某一个设备触发了事件或者超过了指定的等待时间——也就是它们的职责不是做IO,而是帮助调用者寻找当前就绪的设备。同类型的产品是Windows的IOCP,它也是处理多路复用,只是把IO和探测封装在了一起了。

准备的知识有两点:1、fd;2、op->poll。

在Linux里面,设备都被抽象为文件,一系列的设备文件就有自己独立的虚拟文件系统,所以,设备在系统调用参数中的表示就是file description。fd其实就是一个整数(特别地,标准输入,输出,错误输出分别对应的fd是0,1,2)。与内核打交道的时候,传递整数的fd可以在自己的文件系统中作进一步的检查是否合法,如果只是返回指针就不能这样操作了,毕竟指针是无差别无意义的。

通过fd访问file,通过file可以访问其fileOperator,这里面我们要关心的一个fileOp就是poll。因为系统调用poll和select,就是依靠这个文件操作poll实现的。poll文件操作有两个参数,一个是文件本身,一个可以看做是当设备尚未就绪时调用的回调函数,这个函数是把设备自己特有的等待队列传给内核,让内核把当前的进程挂载到其中(因为当设备就绪时,设备就应该去唤醒在自己特有等待队列中的所有节点,这样当前进程就获取了完成的信号了)。poll文件操作返回的必须是一组标准的掩码,其中的各个位指示当前的不同的就绪状态(全0为没有任何事件触发)。

再谈谈早期多路复用的版本poll和select。

本质而言,poll和select的共同点就是,对全部指定设备做一次poll,当然这往往都是还没有就绪的,那就会通过回调函数把当前进程注册到设备的等待队列,如果所有设备返回的掩码都没有显示任何的事件触发,就去掉回调函数的函数指针,进入有限时的睡眠状态,再恢复和不断做poll,再作有限时的睡眠,直到其中一个设备有事件触发为止。只要有事件触发,系统调用返回,回到用户态,用户就可以对相关的fd作进一步的读或者写操作了。当然,这个时候还不是所有的设备都就绪的喔,那就得不断地poll或者select了,而做一次这样的系统调用都得轮询所有的设备,次数是设备数*(睡眠次数-1),也就是时间复杂度是O(n),还得做几次O(n)呢。可见,对于现在普遍的服务器程序,需要同时并发监听数千个连接,并且连接需要重复使用的情况,poll和select就存在这样的性能瓶颈。另外,数千个设备fd在每次调用时,都需要将其从用户空间复制到内核空间,这里的开销不可忽略。

poll和select放在一起,是因为其机制一致,而参数和数据结构就略有不同。select一次性传入三组作用于不同信道的设备fd,分别是输入,输出和错误异常。各组的fd期待各组所特有的,由代码指定的一组事件,如输入信道期待输入就绪,输入挂起和错误等事件。 然后,select就挑选调用者关心的fd做poll文件操作,检测返回的掩码,看看是否有fd所属信道感兴趣的事件,比如看看这个属于输出信道的fd有没有输出就绪等一系列的事件发生,一样地,如果有一个fd发生感兴趣事件就返回调用了。select,为了同时处理三组使用不同的事件判断规则的fd,采用了位图的方式表示,一组一个位图,位长度是当中最大的fd值,上限是1024,三组就是3072,而且这还只是传入的位图,还有一样大小的传出的位图。当fd数越来越多时,所需的存储开销比较大。

既然,一组fd处理起来比较粗放,那就各个fd自己准备好了。poll()系统调用是System V的多元I/O解决方案。它有三个参数,第一个是pollfd结构的数组指针,也就是指向一组fd及其相关信息的指针,因为这个结构包含的除了fd,还有期待的事件掩码和返回的事件掩码,实质上就是将select的中的fd,传入和传出参数归到一个结构之下,也不再把fd分为三组,也不再硬性规定fd感兴趣的事件,这由调用者自己设定。这样,不使用位图来组织数据,也就不需要位图的全部遍历了。按照一般队列地遍历,每个fd做poll文件操作,检查返回的掩码是否有期待的事件,以及做是否有挂起和错误的必要性检查,如果有事件触发,就可以返回调用了。

回到poll和select的共同点,面对高并发多连接的应用情境,它们显现出原来没有考虑到的不足,虽然poll比起select又有所改进了。除了上述的关于每次调用都需要做一次从用户空间到内核空间的拷贝,还有这样的问题,就是当处于这样的应用情境时,poll和select会不得不多次操作,并且每次操作都很有可能需要多次进入睡眠状态,也就是多次全部轮询fd,我们应该怎么处理一些会出现重复而无意义的操作。

这些重复而无意义的操作有:1、从用户到内核空间拷贝,既然长期监视这几个fd,甚至连期待的事件也不会改变,那拷贝无疑就是重复而无意义的,我们可以让内核长期保存所有需要监视的fd甚至期待事件,或者可以再需要时对部分期待事件进行修改;2、将当前线程轮流加入到每个fd对应设备的等待队列,这样做无非是哪一个设备就绪时能够通知进程退出调用,聪明的开发者想到,那就找个“代理”的回调函数,代替当前进程加入fd的等待队列好了(这也是我后来才总结出来,Linux的等待队列,实质上是回调函数队列吧,也可以使用宏来将当前进程“加入”等待队列,其实就是将唤醒当前进程的回调函数加入队列)。这样,像poll系统调用一样,做poll文件操作发现尚未就绪时,它就调用传入的一个回调函数,这是epoll指定的回调函数,它不再像以前的poll系统调用指定的回调函数那样,而是就将那个“代理”的回调函数加入设备的等待队列就好了,这个代理的回调函数就自己乖乖地等待设备就绪时将它唤醒,然后它就把这个设备fd放到一个指定的地方,同时唤醒可能在等待的进程,到这个指定的地方取fd就好了。我们把1和2结合起来就可以这样做了,只拷贝一次fd,一旦确定了fd就可以做poll文件操作,如果有事件当然好啦,马上就把fd放到指定的地方,而通常都是没有的,那就给这个fd的等待队列加一个回调函数,有事件就自动把fd放到指定的地方,当前进程不需要再一个个poll和睡眠等待了。

epoll机制就是这样改进的了。诚然,fd少的时候,当前进程一个个地等问题不大,可是现在和尚多了,方丈就不好管了。以前设备事件触发时,只负责唤醒当前进程就好了,而当前进程也只能傻傻地在poll里面等待或者循环,再来一次poll,也不知道这个由设备提供的poll性能如何,能不能检查出当前进程已经在等待了就立即返回,当然,我也不明白为什么做了一遍的poll之后,去掉回调函数指针了,还得再做,不是说好了会去唤醒进程的吗?

现在就让事件触发回调函数多做一步。本来设备还没就绪就调用一个回调函数了,现在再在这个回调函数里面做一个注册另一个回调函数的操作,目的就是使得设备事件触发多走一步,不仅仅是唤醒当前进程,还要把自己的fd放到指定的地方。就像收本子的班长,以前得一个个学生地去问有没有本子,如果没有,它还得等待一段时间而后又继续问,现在好了,只走一次,如果没有本子,班长就告诉大家去那里交本子,当班长想起要取本子,就去那里看看或者等待一定时间后离开,有本子到了就叫醒他,然后取走。这个道理很简单,就是老师和班干们常说的,大家多做一点工作,我的工作就轻松很多了,尤其是需要管理的东西越来越多时。

这种机制或者说模式,我想在Java的FutureTask里面应该也会用到的,一堆在线程池里面跑着的线程(当然这是任务,不是线程,接口是Callable,不是Runnable.run,是Callable.call,它是可以返回结果的),谁先做好就应该先处理呀,可是难道得一个个问吗?干脆就谁好了,谁就按照既定的操作暴露自己,这样FutureTask的get方法就可以马上知道当前最先完成的线程了,就可以取此线程返回结果了。

epoll由三个系统调用组成,分别是epoll_create,epoll_ctl和epoll_wait。epoll_create用于创建和初始化一些内部使用的数据结构;epoll_ctl用于添加,删除或者修改指定的fd及其期待的事件,epoll_wait就是用于等待任何先前指定的fd事件。

关于epoll内部的数据结构,我就不能详细了解了。

下面就是分别实现一个 demo,然后用 ab 去压力测试,每回都是 10W 个请求,根据不同的并发量测试

单进程单线程

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#define BUFFER_SIZE 4096

int tcpSrv(int listen_port);

int main(int argc, char** argv) {
    tcpSrv(8080);
    return 0;
}

int tcpSrv(int listen_port) {
    int listenfd, connfd;
    struct sockaddr_in servaddr;
    char buff[BUFFER_SIZE+1];
    int n;

    if ((listenfd = socket(AF_INET, SOCK_STREAM, 0)) == -1) {
        printf("create socket error: %s(errno: %d)\n", strerror(errno), errno);
        return -1;
    }

    memset(&servaddr, 0, sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
    servaddr.sin_port = htons(listen_port);

    if (bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr)) == -1) {
        printf("bind socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    if (listen(listenfd, 2048) == -1) {
        printf("listen socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    printf("-----waiting for client's request-----\n");
    while (1) {
        if ((connfd = accept(listenfd, (struct sockaddr *)NULL, NULL)) == -1) {
            printf("accept socket error: %s(errno: %d)\n", strerror(errno), errno);
            close(listenfd);
            return -1;
        }
        memset(&buff, 0, sizeof(buff));
        recv(connfd, buff, BUFFER_SIZE, 0);
        //printf("recv msg:\n%s\n", buff);
        char resp[] = "HTTP/1.0 200 OK\r\n\r\n<html>hello</html>";
        send(connfd, resp, strlen(resp), 0);
        close(connfd);
    }
    
    close(listenfd);
    return 0;
}

一开始觉得这种简单粗暴的方式肯定不行,结果发现还别说,这东西的处理能力还是可以的,看测试结果

单并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      1
Time taken for tests:   7.210 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    13869.50 [#/sec] (mean)
Time per request:       0.072 [ms] (mean)
Time per request:       0.072 [ms] (mean, across all concurrent requests)
Transfer rate:          501.14 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       4
Processing:     0    0   0.0      0       4
Waiting:        0    0   0.0      0       4
Total:          0    0   0.0      0       4

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%      4 (longest request)
linux:~/ab/httpd-2.4.3/support$ 

10 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 10 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      10
Time taken for tests:   5.498 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    18186.97 [#/sec] (mean)
Time per request:       0.550 [ms] (mean)
Time per request:       0.055 [ms] (mean, across all concurrent requests)
Transfer rate:          657.15 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       6
Processing:     0    0   0.1      0       6
Waiting:        0    0   0.1      0       6
Total:          0    1   0.1      1       6

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%      6 (longest request)
linux:~/ab/httpd-2.4.3/support$ 

100 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 100 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      100
Time taken for tests:   5.816 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    17194.97 [#/sec] (mean)
Time per request:       5.816 [ms] (mean)
Time per request:       0.058 [ms] (mean, across all concurrent requests)
Transfer rate:          621.30 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.6      1      11
Processing:     1    5   0.8      5      15
Waiting:        0    2   0.6      2      12
Total:          2    6   0.6      6      15

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      6
  75%      6
  80%      6
  90%      6
  95%      6
  98%      7
  99%      7
 100%     15 (longest request)
linux:~/ab/httpd-2.4.3/support$ 

150 个并发的时候就不行了

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 150 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 99956 requests completed
linux:~/ab/httpd-2.4.3/support$ 

接下来是多进程模型

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <signal.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#define BUFFER_SIZE 4096

int tcpSrv(int listen_port);

int main(int argc, char** argv) {
    tcpSrv(8080);
    return 0;
}

int tcpSrv(int listen_port) {
    int listenfd, connfd;
    struct sockaddr_in servaddr;
    char buff[BUFFER_SIZE+1];
    int n;

    if ((listenfd = socket(AF_INET, SOCK_STREAM, 0)) == -1) {
        printf("create socket error: %s(errno: %d)\n", strerror(errno), errno);
        return -1;
    }

    memset(&servaddr, 0, sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
    servaddr.sin_port = htons(listen_port);

    if (bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr)) == -1) {
        printf("bind socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    if (listen(listenfd, 2048) == -1) {
        printf("listen socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    printf("-----waiting for client's request-----\n");
    while (1) {
        if ((connfd = accept(listenfd, (struct sockaddr *)NULL, NULL)) == -1) {
            printf("accept socket error: %s(errno: %d)\n", strerror(errno), errno);
            close(listenfd);
            return -1;
        }
        signal(SIGCHLD, SIG_IGN);
        pid_t pid = fork();
        if (pid < 0) {
            printf("fork error: %s(errno: %d)\n", strerror(errno), errno);
            close(listenfd);
            return -1;
        } else if (pid == 0) {
            //printf("in child process\n");
            memset(&buff, 0, sizeof(buff));
            recv(connfd, buff, BUFFER_SIZE, 0);
            //printf("recv msg:\n%s\n", buff);
            char resp[] = "HTTP/1.0 200 OK\r\n\r\n<html>hello</html>";
            send(connfd, resp, strlen(resp), 0);
            close(connfd);
            exit(0);
        } else if (pid > 0) {
            //printf("in parent process\n");
            close(connfd);
            continue;
        }
    }
    
    close(listenfd);
    return 0;
}

单并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      1
Time taken for tests:   41.408 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    2415.02 [#/sec] (mean)
Time per request:       0.414 [ms] (mean)
Time per request:       0.414 [ms] (mean, across all concurrent requests)
Transfer rate:          87.26 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     0    0   0.1      0       9
Waiting:        0    0   0.1      0       9
Total:          0    0   0.1      0       9

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%      9 (longest request)
linux:~/ab/httpd-2.4.3/support$ 

10 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 10 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      10
Time taken for tests:   39.517 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    2530.53 [#/sec] (mean)
Time per request:       3.952 [ms] (mean)
Time per request:       0.395 [ms] (mean, across all concurrent requests)
Transfer rate:          91.44 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       3
Processing:     0    4   0.4      4      64
Waiting:        0    4   0.4      4      64
Total:          1    4   0.4      4      64

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%      4
  98%      4
  99%      4
 100%     64 (longest request)
linux:~/ab/httpd-2.4.3/support$ 

100 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 100 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      100
Time taken for tests:   39.980 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    2501.28 [#/sec] (mean)
Time per request:       39.980 [ms] (mean)
Time per request:       0.400 [ms] (mean, across all concurrent requests)
Transfer rate:          90.38 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.3      0      17
Processing:     0   40   1.7     40     120
Waiting:        0   40   1.7     40     120
Total:          3   40   1.6     40     120

Percentage of the requests served within a certain time (ms)
  50%     40
  66%     40
  75%     41
  80%     41
  90%     41
  95%     42
  98%     43
  99%     44
 100%    120 (longest request)
linux:~/ab/httpd-2.4.3/support$ 

500 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 500 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      500
Time taken for tests:   63.222 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    1581.72 [#/sec] (mean)
Time per request:       316.112 [ms] (mean)
Time per request:       0.632 [ms] (mean, across all concurrent requests)
Transfer rate:          57.15 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   38 520.0      0   21004
Processing:     3  208 2942.9     51   60202
Waiting:        0  208 2942.9     51   60202
Total:         24  247 3153.2     51   63206

Percentage of the requests served within a certain time (ms)
  50%     51
  66%     52
  75%     52
  80%     52
  90%     53
  95%     54
  98%     55
  99%     61
 100%  63206 (longest request)
linux:~/ab/httpd-2.4.3/support$ 

1000 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1000 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /
Document Length:        18 bytes

Concurrency Level:      1000
Time taken for tests:   71.625 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    1396.16 [#/sec] (mean)
Time per request:       716.252 [ms] (mean)
Time per request:       0.716 [ms] (mean, across all concurrent requests)
Transfer rate:          50.45 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   68 628.8      0   21002
Processing:     1  442 4622.4     51   68587
Waiting:        0  442 4622.3     51   68587
Total:         17  510 4934.8     51   71588

Percentage of the requests served within a certain time (ms)
  50%     51
  66%     52
  75%     52
  80%     52
  90%     54
  95%     55
  98%     59
  99%   4107
 100%  71588 (longest request)
linux:~/ab/httpd-2.4.3/support$ 

再来是多线程

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <unistd.h>
#include <pthread.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>

#define BUFFER_SIZE 4096

int tcpSrv(int listen_port);
void* process_thread(void* sockfd);

int main(int argc, char** argv) {
    tcpSrv(8080);
    return 0;
}

void* process_thread(void* sockfd) {
    //printf("thread created\n");
    int connfd = *((int *)sockfd);
    char buff[BUFFER_SIZE+1];
    memset(&buff, 0, sizeof(buff));
    recv(connfd, buff, BUFFER_SIZE, 0);
    //printf("recv msg:\n%s\n", buff);
    char resp[] = "HTTP/1.0 200 OK\r\n\r\n<html>hello</html>";
    send(connfd, resp, strlen(resp), 0);
    close(connfd);
    //printf("thread about to exit\n");
    pthread_exit(NULL);
}

int tcpSrv(int listen_port) {
    int listenfd, connfd;
    struct sockaddr_in servaddr;

    if ((listenfd = socket(AF_INET, SOCK_STREAM, 0)) == -1) {
        printf("create socket error: %s(errno: %d)\n", strerror(errno), errno);
        return -1;
    }

    memset(&servaddr, 0, sizeof(servaddr));
    servaddr.sin_family = AF_INET;
    servaddr.sin_addr.s_addr = htonl(INADDR_ANY);
    servaddr.sin_port = htons(listen_port);

    if (bind(listenfd, (struct sockaddr *)&servaddr, sizeof(servaddr)) == -1) {
        printf("bind socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    if (listen(listenfd, 2048) == -1) {
        printf("listen socket error: %s(errno: %d)\n", strerror(errno), errno);
        close(listenfd);
        return -1;
    }

    printf("-----waiting for client's request-----\n");
    while (1) {
        if ((connfd = accept(listenfd, (struct sockaddr *)NULL, NULL)) == -1) {
            printf("accept socket error: %s(errno: %d)\n", strerror(errno), errno);
            close(listenfd);
            return -1;
        }
        //printf("main thread about to create thread\n");
        pthread_t tid;
        pthread_attr_t opt;
        pthread_attr_init(&opt);
        pthread_attr_setdetachstate(&opt, PTHREAD_CREATE_DETACHED);
        if (pthread_create(&tid, &opt, process_thread, &connfd) != 0) {
            printf("error line: %d\t", __LINE__);
            perror(__FILE__);
            return -1;
        }
        pthread_attr_destroy(&opt);
        //printf("main thread finish creating thread\n");
    }
    
    close(listenfd);
    return 0;
}

单并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /123
Document Length:        18 bytes

Concurrency Level:      1
Time taken for tests:   7.629 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    13108.37 [#/sec] (mean)
Time per request:       0.076 [ms] (mean)
Time per request:       0.076 [ms] (mean, across all concurrent requests)
Transfer rate:          473.64 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       4
Processing:     0    0   0.1      0      24
Waiting:        0    0   0.1      0      23
Total:          0    0   0.1      0      24

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%     24 (longest request)
linux:~/ab/httpd-2.4.3/support$ 

10 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 10 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
apr_pollset_poll: The timeout specified has expired (70007)
Total of 99991 requests completed
linux:~/ab/httpd-2.4.3/support$ 

至于 select 和 poll 就没有写了,直接写 epoll 模型

#include <iostream>
#include <sys/socket.h>
#include <sys/epoll.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <errno.h>

using namespace std;

#define MAX_LINE 1024
#define OPEN_MAX 100
#define LISTENQ 1024
#define SERV_PORT 8080
#define INFTIM 1000
#define MAX_EVENTS 1024
#define EP_TIMEOUT 500

int tcpSrv(int listen_port);
int set_non_block(int fd);

int main(int argc, char* argv[]) {
    tcpSrv(SERV_PORT);
    return 0;
}

int set_non_block(int fd) {
    int fd_fl = fcntl(fd, F_GETFL); 
    fd_fl |= O_NONBLOCK;
    if (fcntl(fd, F_SETFL, fd_fl) < 0) {
        printf("%d\n:", __LINE__);perror(__FILE__);
        return -1;
    }
    return 0;
}

int tcpSrv(int listen_port) {
    int listenfd = socket(AF_INET, SOCK_STREAM, 0);
    if (-1 == listenfd) {
        printf("%d\n:", __LINE__);perror(__FILE__);
        return -1;
    }
    set_non_block(listenfd);

    struct epoll_event ev;
    ev.data.fd = listenfd;
    ev.events = EPOLLIN|EPOLLET;
    int epfd = epoll_create(4096);
    epoll_ctl(epfd, EPOLL_CTL_ADD, listenfd, &ev);

    struct sockaddr_in serveraddr;
    bzero(&serveraddr, sizeof(serveraddr));
    serveraddr.sin_family = AF_INET;
    char* local_addr = "127.0.0.1";
    inet_aton(local_addr, &(serveraddr.sin_addr));
    serveraddr.sin_port = htons(listen_port);

    if (bind(listenfd, (sockaddr*)&serveraddr, sizeof(serveraddr)) == -1) {
        printf("%d\n:", __LINE__);perror(__FILE__);
        close(listenfd);
        return -1;
    }
    
    if (listen(listenfd, LISTENQ) == -1) {
        printf("%d\n:", __LINE__);perror(__FILE__);
        close(listenfd);
        return -1;
    }

    struct epoll_event events[MAX_EVENTS];
    //struct sockaddr_in clientaddr;
    //socklen_t clilen;
    while (1) {
        int nfds = epoll_wait(epfd, events, MAX_EVENTS, EP_TIMEOUT);
        //printf("events count:%d\n", nfds);
        for (int i=0; i<nfds; i++) {
            if (events[i].data.fd == listenfd) {
                while (1) {
                    int connfd = accept(listenfd, NULL, NULL);
                    if (connfd < 0) {
                        //printf("%d\n:", __LINE__);perror(__FILE__);
                        break;
                    }
                    //char *str_client_addr = inet_ntoa(clientaddr.sin_addr);
                    //cout<<"accept a connection from: "<<str_client_addr<<endl;
                    ev.data.fd = connfd;
                    ev.events = EPOLLIN | EPOLLET;
                    epoll_ctl(epfd, EPOLL_CTL_ADD, connfd, &ev);
                }
            } else if (events[i].events & EPOLLIN) {
                //cout<<"EPOLLIN"<<endl;
                int sockfd = events[i].data.fd;
                if (sockfd < 0) {
                    printf("%d\n:", __LINE__);perror(__FILE__);
                    continue;
                }
                char buffer[MAX_LINE+1];
                memset(buffer, 0, sizeof(buffer));
                read(sockfd, buffer, MAX_LINE);
                //printf("%s\n", buffer);
                set_non_block(sockfd);
                char html_resp[] = "HTTP/1.1 200 OK\r\n\r\n<html>epoll</html>";
                write(sockfd, html_resp, strlen(html_resp));
                epoll_ctl(epfd, EPOLL_CTL_DEL, sockfd, NULL);
                close(sockfd);
            }
        }
    }
    
    return 0;
}

单并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /123
Document Length:        18 bytes

Concurrency Level:      1
Time taken for tests:   7.474 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    13380.33 [#/sec] (mean)
Time per request:       0.075 [ms] (mean)
Time per request:       0.075 [ms] (mean, across all concurrent requests)
Transfer rate:          483.47 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       4
Processing:     0    0   0.0      0       5
Waiting:        0    0   0.0      0       5
Total:          0    0   0.0      0       5

Percentage of the requests served within a certain time (ms)
  50%      0
  66%      0
  75%      0
  80%      0
  90%      0
  95%      0
  98%      0
  99%      0
 100%      5 (longest request)

10 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 10 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /123
Document Length:        18 bytes

Concurrency Level:      10
Time taken for tests:   5.991 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    16691.90 [#/sec] (mean)
Time per request:       0.599 [ms] (mean)
Time per request:       0.060 [ms] (mean, across all concurrent requests)
Transfer rate:          603.13 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       6
Processing:     0    0   0.1      0       7
Waiting:        0    0   0.1      0       6
Total:          0    1   0.2      1       7

Percentage of the requests served within a certain time (ms)
  50%      1
  66%      1
  75%      1
  80%      1
  90%      1
  95%      1
  98%      1
  99%      1
 100%      7 (longest request)

100 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 100 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /123
Document Length:        18 bytes

Concurrency Level:      100
Time taken for tests:   6.233 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    16043.99 [#/sec] (mean)
Time per request:       6.233 [ms] (mean)
Time per request:       0.062 [ms] (mean, across all concurrent requests)
Transfer rate:          579.71 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   1.0      1       9
Processing:     1    5   0.9      5      12
Waiting:        1    2   0.9      2      10
Total:          2    6   1.4      6      15

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      7
  75%      7
  80%      7
  90%      8
  95%      8
  98%     10
  99%     10
 100%     15 (longest request)

1000 个并发

linux:~/ab/httpd-2.4.3/support$ ./ab -n 100000 -c 1000 http://localhost:8080/123
This is ApacheBench, Version 2.3 <$Revision: 1373084 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        
Server Hostname:        localhost
Server Port:            8080

Document Path:          /123
Document Length:        18 bytes

Concurrency Level:      1000
Time taken for tests:   7.099 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      3700000 bytes
HTML transferred:       1800000 bytes
Requests per second:    14087.17 [#/sec] (mean)
Time per request:       70.987 [ms] (mean)
Time per request:       0.071 [ms] (mean, across all concurrent requests)
Transfer rate:          509.01 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1   26 115.1     19    3054
Processing:    16   44   9.7     43      76
Waiting:        1   18   8.0     18      54
Total:         20   70 115.7     65    3103

Percentage of the requests served within a certain time (ms)
  50%     65
  66%     73
  75%     79
  80%     82
  90%     89
  95%     95
  98%    101
  99%    106
 100%   3103 (longest request)
linux:~/ab/httpd-2.4.3/support$ 

可以看到 epoll 确实是很猛的,这里有一个地方需要注意,我调了两天,就是在 epoll 的 EPOLLIN 事件发生在 listenfd 上的时候,这个时候需要去 accept,这个应该都没有什么问题,但是我一开始只是 accept 了一个,而实际上,这个时候,是可能有多个连接在等待 accept 的,于是就导致了并发能力很不行,正确的做法应该是一直 accept,直到没有东西可以 accept 了,这样才能有并发能力。

Leave a Reply

Your email address will not be published. Required fields are marked *