我们有一台大型SSD存储服务器,其中10 Gbps专用接口连接到10台小型服务器(内部网络),每台服务器具有1 Gbps专用接口和1 Gbps公用接口,它们通过Nginx代理作为存储的客户端。
用于将大型文件下载到客户端的存储的主要用途; 500MB〜3GB文件。
我们注意到,大型服务器的IO读取速度非常高,大约为3.5 GB / s(〜28 Gbps),而专用网络接口仅向小型服务器发送了大约6 Gbps。
我对proxy_buffering,proxy_buffers和output_buffers进行了更改,但是没有效果。
使用nginx代理处理大型文件是否正常? 是否有解决方案可以减少磁盘IO,使其与客户端真正消耗的磁盘IO一样?
小型服务器上的Nginx代理配置:
proxy_cache off;
proxy_max_temp_file_size 0;
proxy_request_buffering on;
proxy_buffering on;
proxy_buffer_size 512k;
proxy_buffers 1 512k;
大型服务器上的Nginx配置(源):
worker_processes auto;
worker_rlimit_nofile 10000;
thread_pool default threads=64 max_queue=65536;
events {
worker_connections 32768;
multi_accept on;
accept_mutex off;
use epoll;
}
http {
tcp_nopush on;
tcp_nodelay on;
client_body_timeout 12;
client_header_timeout 12;
keepalive_timeout 15;
send_timeout 10;
aio threads;
sendfile on;
sendfile_max_chunk 512k;
open_file_cache max=30000 inactive=60s;
output_buffers 1 512k;
}
iotop:
Total DISK READ : 3.06 G/s | Total DISK WRITE : 0.00 B/s
Actual DISK READ: 3.052.82 | Actual DISK WRITE: 1006.26 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
172187 be/4 nginx 354.04 M/s 0.00 B/s 0.00 % 16.68 % nginx: worker process
172057 be/4 nginx 169.40 M/s 0.00 B/s 0.00 % 12.48 % nginx: worker process
171642 be/4 nginx 216.52 M/s 0.00 B/s 0.00 % 11.94 % nginx: worker process
172139 be/4 nginx 126.93 M/s 0.00 B/s 0.00 % 7.67 % nginx: worker process
171599 be/4 nginx 175.99 M/s 0.00 B/s 0.00 % 6.95 % nginx: worker process
171407 be/4 nginx 124.73 M/s 0.00 B/s 0.00 % 5.28 % nginx: worker process
171385 be/4 nginx 2.18 M/s 0.00 B/s 0.00 % 3.94 % nginx: worker process
172603 be/4 nginx 3.65 M/s 0.00 B/s 0.00 % 2.34 % nginx: worker process
172659 be/4 nginx 898.32 K/s 0.00 B/s 0.00 % 2.21 % nginx: worker process
171602 be/4 nginx 24.44 M/s 0.00 B/s 0.00 % 2.17 % nginx: worker process
172874 be/4 nginx 3.35 M/s 0.00 B/s 0.00 % 2.05 % nginx: worker process
172801 be/4 nginx 3.50 M/s 0.00 B/s 0.00 % 1.78 % nginx: worker process
172309 be/4 nginx 14.12 M/s 0.00 B/s 0.00 % 1.67 % nginx: worker process
172326 be/4 nginx 3.74 M/s 0.00 B/s 0.00 % 1.65 % nginx: worker process
172870 be/4 nginx 2.14 M/s 0.00 B/s 0.00 % 1.58 % nginx: worker process
172173 be/4 nginx 6.53 M/s 0.00 B/s 0.00 % 1.54 % nginx: worker process
172360 be/4 nginx 22.98 M/s 0.00 B/s 0.00 % 1.50 % nginx: worker process
171219 be/4 nginx 45.75 M/s 0.00 B/s 0.00 % 1.49 % nginx: worker process
172794 be/4 nginx 12.21 M/s 0.00 B/s 0.00 % 1.45 % nginx: worker process
172127 be/4 nginx 449.16 K/s 0.00 B/s 0.00 % 1.42 % nginx: worker process
172511 be/4 nginx 1239.55 K/s 0.00 B/s 0.00 % 1.35 % nginx: worker process
172709 be/4 nginx 5.84 M/s 0.00 B/s 0.00 % 1.33 % nginx: worker process
172337 be/4 nginx 4.94 M/s 0.00 B/s 0.00 % 1.33 % nginx: worker process
172122 be/4 nginx 4.80 M/s 0.00 B/s 0.00 % 1.31 % nginx: worker process
172617 be/4 nginx 4.65 M/s 0.00 B/s 0.00 % 1.29 % nginx: worker process
172857 be/4 nginx 17.41 M/s 0.00 B/s 0.00 % 1.26 % nginx: worker process
172530 be/4 nginx 449.16 K/s 0.00 B/s 0.00 % 1.26 % nginx: worker process
172775 be/4 nginx 3.25 M/s 0.00 B/s 0.00 % 1.26 % nginx: worker process
172295 be/4 nginx 685.93 K/s 0.00 B/s 0.00 % 1.25 % nginx: worker process
172840 be/4 nginx 5.41 M/s 0.00 B/s 0.00 % 1.24 % nginx: worker process
172827 be/4 nginx 21.82 M/s 0.00 B/s 0.00 % 1.23 % nginx: worker process
172633 be/4 nginx 29.20 M/s 0.00 B/s 0.00 % 1.18 % nginx: worker process
172137 be/4 nginx 449.16 K/s 0.00 B/s 0.00 % 1.14 % nginx: worker process
172348 be/4 nginx 7.21 M/s 0.00 B/s 0.00 % 1.13 % nginx: worker process
iostat:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
md0 0.00 0.00 13177.50 0.00 3144810.00 0.00 477.30 0.00 0.00 0.00 0.00 0.00 0.00