请求计数增加最终导致“错误：套接字挂起”

Question

当请求数量增加时，后端服务/ nginx代理开始响应“错误：套接字挂起”。设置如下。

OS：CentOS 6

Express JS服务-> nginx作为代理-> Gunicorn运行的flask应用程序

JS应用程序同时向另一个服务发送多个请求，当请求计数超过〜100时，它将开始返回错误响应。如果计数较低，则一切正常。

我已经按照Gunicorn文档中的nginx的示例配置+增加了超时限制+增加了nginx打开文件限制。我也尝试过keepalive选项，但问题仍然存在。 Gunicorn没有显示任何错误。

nginx配置片段：

upstream app_server {
    server 127.0.0.1:8000 fail_timeout=0;
    keepalive 100;
}

server {
    listen 5001;
    client_max_body_size 4G;

    keepalive_timeout 300;

    root /path/to/app/current/public; # static files

    location / {
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        # Timeouts
        proxy_read_timeout 300;
        proxy_connect_timeout 300;
        proxy_send_timeout 300;
        send_timeout 300;

        proxy_http_version 1.1;

        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://app_server;
    }
}

从代理收到错误响应：

{ RequestError: Error: socket hang up
    at new RequestError (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/errors.js:14:15)
    at Request.plumbing.callback (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/plumbing.js:87:29)
    at Request.RP$callback [as _callback] (/home/pm2deploy/apps/app-backend/source/node_modules/request-promise-core/lib/plumbing.js:46:31)
    at self.callback (/home/pm2deploy/apps/app-backend/source/node_modules/request/request.js:185:22)
    at Request.emit (events.js:160:13)
    at Request.onRequestError (/home/pm2deploy/apps/app-backend/source/node_modules/request/request.js:881:8)
    at ClientRequest.emit (events.js:160:13)
    at Socket.socketOnEnd (_http_client.js:423:9)
    at Socket.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19)
  name: 'RequestError',
  message: 'Error: socket hang up',
  cause: { Error: socket hang up
    at createHangUpError (_http_client.js:330:15)
    at Socket.socketOnEnd (_http_client.js:423:23)
    at Socket.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19) code: 'ECONNRESET' },
  error: { Error: socket hang up
    at createHangUpError (_http_client.js:330:15)
    at Socket.socketOnEnd (_http_client.js:423:23)
    at Socket.emit (events.js:165:20)
    at endReadableNT (_stream_readable.js:1101:12)
    at process._tickCallback (internal/process/next_tick.js:152:19) code: 'ECONNRESET' },
  options:
   { method: 'PUT',
     uri: 'http://localhost:5001/transformers/segmentAvg',
     qs:
      { stdMultiplier: 2,
        segmentLeft: 1509366682333,
        segmentRight: 1509367401685 },
     body: { index: [Array], values: [Array] },
     headers: {},
     json: true,
     callback: [Function: RP$callback],
     transform: undefined,
     simple: true,
     resolveWithFullResponse: false,
     transform2xxOnly: false },
  response: undefined }

已添加：

在OS日志中记录了以下条目：

possible SYN flooding on port X. Sending cookies.

Answer 1

内核套接字积压已达到限制，并在请求后放弃。

原因：由于Red Hat Enterprise Linux中的LISTEN套接字缓冲区已满，内核丢弃了TCP连接

增加内核套接字积压限制

检查当前值：

# sysctl net.core.somaxconn
net.core.somaxconn = 128

增加值：

# sysctl -w net.core.somaxconn=2048
net.core.somaxconn = 2048

再次查看以确认更改：

# sysctl net.core.somaxconn
net.core.somaxconn = 2048

坚持更改：

echo "net.core.somaxconn = 2048" >> /etc/sysctl.conf

增加应用程序套接字侦听积压

uWSGI的配置参数

listen=1024

---

此解决方案来自https://access.redhat.com/solutions/30453