Nginx代理高IO读取

时间:2018-07-23 17:26:32

标签: nginx proxy io nginx-reverse-proxy

我们有一台大型SSD存储服务器,其中10 Gbps专用接口连接到10台小型服务器(内部网络),每台服务器具有1 Gbps专用接口和1 Gbps公用接口,它们通过Nginx代理作为存储的客户端。

用于将大型文件下载到客户端的存储的主要用途; 500MB〜3GB文件。

我们注意到,大型服务器的IO读取速度非常高,大约为3.5 GB / s(〜28 Gbps),而专用网络接口仅向小型服务器发送了大约6 Gbps。

我对proxy_buffering,proxy_buffers和output_buffers进行了更改,但是没有效果。

使用nginx代理处理大型文件是否正常? 是否有解决方案可以减少磁盘IO,使其与客户端真正消耗的磁盘IO一样?

小型服务器上的Nginx代理配置:

proxy_cache                 off;
proxy_max_temp_file_size    0;
proxy_request_buffering     on;
proxy_buffering             on;
proxy_buffer_size           512k;
proxy_buffers               1   512k;

大型服务器上的Nginx配置(源):

worker_processes  auto;
worker_rlimit_nofile 10000;

thread_pool default threads=64 max_queue=65536;

events {
    worker_connections  32768;
    multi_accept on;
    accept_mutex off;
    use epoll;
}

http {
    tcp_nopush     on;
    tcp_nodelay     on;

    client_body_timeout 12;
    client_header_timeout 12;
    keepalive_timeout 15;
    send_timeout 10;

    aio                     threads;
    sendfile                on;
    sendfile_max_chunk      512k;
    open_file_cache         max=30000   inactive=60s;
    output_buffers          1  512k;
}

iotop:

Total DISK READ :       3.06 G/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:       3.052.82 | Actual DISK WRITE:    1006.26 K/s
   TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
172187 be/4 nginx     354.04 M/s    0.00 B/s  0.00 % 16.68 % nginx: worker process
172057 be/4 nginx     169.40 M/s    0.00 B/s  0.00 % 12.48 % nginx: worker process
171642 be/4 nginx     216.52 M/s    0.00 B/s  0.00 % 11.94 % nginx: worker process
172139 be/4 nginx     126.93 M/s    0.00 B/s  0.00 %  7.67 % nginx: worker process
171599 be/4 nginx     175.99 M/s    0.00 B/s  0.00 %  6.95 % nginx: worker process
171407 be/4 nginx     124.73 M/s    0.00 B/s  0.00 %  5.28 % nginx: worker process
171385 be/4 nginx       2.18 M/s    0.00 B/s  0.00 %  3.94 % nginx: worker process
172603 be/4 nginx       3.65 M/s    0.00 B/s  0.00 %  2.34 % nginx: worker process
172659 be/4 nginx     898.32 K/s    0.00 B/s  0.00 %  2.21 % nginx: worker process
171602 be/4 nginx      24.44 M/s    0.00 B/s  0.00 %  2.17 % nginx: worker process
172874 be/4 nginx       3.35 M/s    0.00 B/s  0.00 %  2.05 % nginx: worker process
172801 be/4 nginx       3.50 M/s    0.00 B/s  0.00 %  1.78 % nginx: worker process
172309 be/4 nginx      14.12 M/s    0.00 B/s  0.00 %  1.67 % nginx: worker process
172326 be/4 nginx       3.74 M/s    0.00 B/s  0.00 %  1.65 % nginx: worker process
172870 be/4 nginx       2.14 M/s    0.00 B/s  0.00 %  1.58 % nginx: worker process
172173 be/4 nginx       6.53 M/s    0.00 B/s  0.00 %  1.54 % nginx: worker process
172360 be/4 nginx      22.98 M/s    0.00 B/s  0.00 %  1.50 % nginx: worker process
171219 be/4 nginx      45.75 M/s    0.00 B/s  0.00 %  1.49 % nginx: worker process
172794 be/4 nginx      12.21 M/s    0.00 B/s  0.00 %  1.45 % nginx: worker process
172127 be/4 nginx     449.16 K/s    0.00 B/s  0.00 %  1.42 % nginx: worker process
172511 be/4 nginx    1239.55 K/s    0.00 B/s  0.00 %  1.35 % nginx: worker process
172709 be/4 nginx       5.84 M/s    0.00 B/s  0.00 %  1.33 % nginx: worker process
172337 be/4 nginx       4.94 M/s    0.00 B/s  0.00 %  1.33 % nginx: worker process
172122 be/4 nginx       4.80 M/s    0.00 B/s  0.00 %  1.31 % nginx: worker process
172617 be/4 nginx       4.65 M/s    0.00 B/s  0.00 %  1.29 % nginx: worker process
172857 be/4 nginx      17.41 M/s    0.00 B/s  0.00 %  1.26 % nginx: worker process
172530 be/4 nginx     449.16 K/s    0.00 B/s  0.00 %  1.26 % nginx: worker process
172775 be/4 nginx       3.25 M/s    0.00 B/s  0.00 %  1.26 % nginx: worker process
172295 be/4 nginx     685.93 K/s    0.00 B/s  0.00 %  1.25 % nginx: worker process
172840 be/4 nginx       5.41 M/s    0.00 B/s  0.00 %  1.24 % nginx: worker process
172827 be/4 nginx      21.82 M/s    0.00 B/s  0.00 %  1.23 % nginx: worker process
172633 be/4 nginx      29.20 M/s    0.00 B/s  0.00 %  1.18 % nginx: worker process
172137 be/4 nginx     449.16 K/s    0.00 B/s  0.00 %  1.14 % nginx: worker process
172348 be/4 nginx       7.21 M/s    0.00 B/s  0.00 %  1.13 % nginx: worker process

iostat:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
md0               0.00     0.00 13177.50    0.00 3144810.00     0.00   477.30     0.00    0.00    0.00    0.00   0.00   0.00

1 个答案:

答案 0 :(得分:0)

尝试增加blockdev提前读取,以减少小型服务器上的IO等待。

blockdev --setra 8192 /dev/sda

Details.