要点:
我正在尝试直接将大型文件上传到PHP而没有使用nginx fastcgi缓冲,因此我可以立即处理来自php://input
的上传数据(即作为流)。
详细说明:
我正在寻找的所需行为是让nginx在到达nginx时传递每个字节(或字节块)并将其传递到上游,在这种情况下是php-fpm。原因是能够获得非常大的文件上传并尽快处理它们(即,一旦每个块可用,就将数据移动到S3)。现在,nginx已经添加了以fastcgi_request_buffering
配置选项的形式在1.7.11 http://nginx.org/en/CHANGES中禁用fastcgi缓冲的支持。
实现此功能时,似乎nginx不遵守此指令,或者php-fpm不会将数据流式传输给worker。以下是卷曲请求的结果:
[root@localhost app]# ll -h large.txt
-rw-r--r--. 1 nginx nginx 307M Nov 10 2015 large.txt
[root@localhost app]# time curl -vs -XPOST --data-binary @large.txt http://127.0.0.1:80
* About to connect() to 127.0.0.1 port 80 (#0)
* Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1
> Accept: */*
> Content-Length: 321912832
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 404 Not Found
< Server: nginx
< Date: Fri, 06 Nov 2015 19:18:27 GMT
< Content-Type: text/html
< Content-Length: 162
< Connection: keep-alive
* HTTP error before end of send, stop sending
<
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Closing connection 0
real 7m43.438s
user 0m0.653s
sys 0m1.219s
[root@localhost app]#
在这样做时,我正在拖尾nginx和php-fpm日志:
==> /var/log/php-fpm/www-error.log <==
[06-Nov-2015 12:18:27 America/Denver] PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 268173441 bytes) in Unknown on line 0
==> /var/log/nginx/error.log <==
2015/11/06 14:18:27 [error] 31657#0: *1 upstream sent unexpected FastCGI record: 3 while reading response header from upstream, client: 127.0.0.1, server: myserver.whatever.com, request: "POST / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1"
2015/11/06 14:18:27 [error] 31657#0: *1 open() "/etc/nginx/html/50x.html" failed (2: No such file or directory), client: 127.0.0.1, server: myserver.whatever.com, request: "POST / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1"
==> /var/log/nginx/access.log <==
127.0.0.1 - - [06/Nov/2015:14:18:27 -0500] "POST / HTTP/1.1" 404 187 "-" "curl/7.29.0"
==> /var/log/php-fpm/error.log <==
[06-Nov-2015 14:18:27] WARNING: [pool www] child 31619 exited with code 70 after 479.683142 seconds from start
[06-Nov-2015 14:18:27] NOTICE: [pool www] child 31809 started
因此,问题很明显:PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 268173441 bytes) in Unknown on line 0
问题是,nginx是否遵守fastcgi_request_buffering off;
指令,或者php-fpm流程管理器搞砸了,而不是直接将数据流式传输到php worker,而是作为整个请求对象传递。
或者,也许我错过了php-fpm的一些配置:http://php.net/manual/en/install.fpm.configuration.php
以下是一些相关的信息:
PHP fpm configs:
[root@localhost nginx]# cat /etc/php-fpm.conf | grep -vE '^;' | grep -v '^$'
include=/etc/php-fpm.d/*.conf
[global]
pid = /run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/error.log
daemonize = yes
pool config:
[root@localhost nginx]# cat /etc/php-fpm.d/www.conf | grep -vE '^;' | grep -v '^$'
[www]
user = apache
group = apache
listen = 127.0.0.1:9000
listen.allowed_clients = 127.0.0.1
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
slowlog = /var/log/php-fpm/www-slow.log
php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on
php_value[session.save_handler] = files
php_value[session.save_path] = /var/lib/php/session
php_value[soap.wsdl_cache_dir] = /var/lib/php/wsdlcache
php_value[upload_max_filesize] = 100G
php_value[post_max_size] = 100G
php_value[max_execution_time] = 3600
request_terminate_timeout = 3600
request_slowlog_timeout = 60
php_value[max_input_time] = 3600
版本:
[root@localhost nginx]# php -v
PHP 5.6.15 (cli) (built: Oct 29 2015 14:18:11)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
with Xdebug v2.3.3, Copyright (c) 2002-2015, by Derick Rethans
[root@localhost nginx]# php-fpm -v
PHP 5.6.15 (fpm-fcgi) (built: Oct 29 2015 14:18:34)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
with Xdebug v2.3.3, Copyright (c) 2002-2015, by Derick Rethans
[root@localhost nginx]# nginx -v
nginx version: nginx/1.8.0
[root@localhost nginx]# uname -a
Linux localhost.localdomain 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost nginx]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)
和nginx配置:
server {
listen 0.0.0.0:80 default_server;
server_name myserver.com;
location / {
root /opt/my/app/public;
index index.html index.htm index.php;
try_files $uri $uri/ /index.php?$args;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
location ~ \.php$ {
root /opt/my/app/public;
try_files $uri = 404;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_request_buffering off;
fastcgi_read_timeout 3600;
include fastcgi_params;
}
}