Question

我有一个非常高流量的Nginx服务器，为一系列网站提供静态内容。我无法弄清楚为什么磁盘写入如此之高。

VMWare ESXi 6.0主机（数据存储位于RAID10中的4x企业级SSD）上运行具有4核和16GB RAM的Ubuntu 14.04.3 LTS VM。 NGINX v 1.4.6

该实例目前正在处理大约75,000个连接（我知道这列出了所有连接，不仅仅是连接，而且TIME_WAIT连接仍然使用端口）

<%!
    public String renderComment(String comment) {
        return comment.replaceAll("<", "&lt;").replaceAll(">", "&gt;").replaceAll("\n", "<br />");
    }
%>

<c:forEach var="answer" items="${answerList}" varStatus="status">
    <%=renderComment(${answer.cn})%> <!-- This is wrong. -->
    <%=renderComment(answer.cn)%> <!-- This is wrong.-->
    ${renderComment(answer.cn)} <!-- This is wrong, too! How can I do? -->
</c:forEach>

推动大约50 MiB

$ netstat -tn | wc -l
75237

写入率是通过屋顶！

$ sudo bmon
Interfaces                     x RX bps       pps     %x TX bps       pps     %
->lo                           x      4B        0      x      4B        0
  eth0                         x   1.60MiB  17.57K     x  52.93MiB  13.57K
    qdisc none (mq)            x      0         0      x  54.64MiB  41.08K
      class :1 (mq)            x      0         0      x  21.49MiB  15.46K
      class :2 (mq)            x      0         0      x  11.65MiB   9.57K
      class :3 (mq)            x      0         0      x  11.62MiB   8.65K
      class :4 (mq)            x      0         0      x   9.88MiB   7.40K

这是我当前的Nginx配置，注释行是我尝试过的变体。

$ sudo iostat
Linux 3.13.0-52-generic (hostname)   11/30/2015      _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.13    0.00    4.80   62.99    0.00   30.08

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sda              93.21       696.88     34015.01    6759664  329942882
dm-0            105.09       696.56     34022.58    6756593  330016284
dm-1              0.02         0.09         0.00        896          0



$ sudo iotop -k -o
Total DISK READ :      71.03 K/s | Total DISK WRITE :   45959.36 K/s
Actual DISK READ:      71.03 K/s | Actual DISK WRITE:   33324.19 K/s
  TID  PRIO  USER     DISK READ DISK WRITE>  SWAPIN      IO    COMMAND
 1128 be/4 www-data    0.00 K/s 12134.03 K/s  0.00 % 78.36 % nginx: worker process
 1119 be/4 www-data    0.00 K/s 8073.57 K/s  0.00 % 89.34 % nginx: worker process
 1109 be/4 www-data   71.03 K/s 6065.04 K/s  0.00 % 26.60 % nginx: worker process
 1110 be/4 www-data    0.00 K/s 4032.84 K/s  0.00 % 89.23 % nginx: worker process
 1105 be/4 www-data    0.00 K/s 2024.31 K/s  0.00 %  0.00 % nginx: worker process
 1113 be/4 www-data    0.00 K/s 2024.31 K/s  0.00 % 20.72 % nginx: worker process
 1115 be/4 www-data    0.00 K/s 2024.31 K/s  0.00 %  0.00 % nginx: worker process
 1120 be/4 www-data    0.00 K/s 2024.31 K/s  0.00 %  0.00 % nginx: worker process
 1121 be/4 www-data    0.00 K/s 2024.31 K/s  0.00 % 61.78 % nginx: worker process
 1114 be/4 www-data    0.00 K/s 2020.37 K/s  0.00 %  0.00 % nginx: worker process
 1106 be/4 www-data    0.00 K/s 2016.42 K/s  0.00 % 48.97 % nginx: worker process
 1122 be/4 www-data    0.00 K/s 1365.32 K/s  0.00 %  0.00 % nginx: worker process
  184 be/3 root        0.00 K/s  126.27 K/s  0.00 % 90.53 % [jbd2/dm-0-8]
 1127 be/4 www-data    0.00 K/s    3.95 K/s  0.00 %  0.00 % nginx: worker process

这些是我对/etc/sysctl.conf

所做的修改

user www-data;
worker_processes 32; # I know the recommended is 1 per core, but with this set to auto, images started breaking instead of just lagging (which my boss thought was preferable short-term)
#worker_processes auto;
worker_rlimit_nofile 100000;
pid /run/nginx.pid;

events {
        worker_connections 4000;
        multi_accept on;
        use epoll;
        # accept_mutex off;
}

http {
        sendfile on;

        tcp_nopush on;
        tcp_nodelay on;

        keepalive_timeout 15;
        keepalive_requests 200;

        reset_timedout_connection on;

        types_hash_max_size 2048;
        server_tokens off;

        open_file_cache max=200000 inactive=20s;
        open_file_cache_valid 30s;
        open_file_cache_min_uses 2;
        open_file_cache_errors on;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        index index.php index.htm index.html;

#        client_body_buffer_size 10k;
#        client_body_buffer_size 16K;
#        client_body_buffer_size 128K;
        client_body_buffer_size 1m;
        client_header_buffer_size 1k;
#        client_header_buffer_size 2k;
        client_max_body_size 25m;
#        large_client_header_buffers 2 1k;
        large_client_header_buffers 4 8k;

        client_body_timeout 15;
        client_header_timeout 15;

        send_timeout 2;

        access_log off;
        error_log /var/log/nginx/error.log crit;

        gzip on;
        gzip_disable "msie6";

        gzip_proxied expired no-cache no-store private auth;
        gzip_comp_level 2;
        gzip_min_length 10240;
        gzip_types text/plain text/css text/xml text/javascript application/json application/x-javascript application/xml application/xml+rss;

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

我对导致高写入率的原因感到茫然。我认为这是因为客户端缓冲区，但没有任何变化产生任何差异。我确保已安装所有更新并在服务器上重新启动，但NOTHING降低了写入速率。任何帮助将不胜感激！

Answer 1

其中一个服务器块正在向一个php脚本发送散列请求，而PHP脚本又根据散列来发送文件。这听起来很精致，直到你深入了解文档并意识到nginx在使用unix套接字时不能使用sendfile性能提升。切换php脚本从发送文件到为文件放置一个X-Accel-Redirect标头，我的写入几乎降为0。

https://www.reddit.com/r/nginx/comments/3v17hq/cant_get_rid_of_high_io_writes_on_nginx_content/

Ubuntu - Nginx - 极高的IO写入

1 个答案: