nginx + php-fpm缓存大文件上传

时间:2015-11-10 15:50:00

标签: php nginx

要点:

我正在尝试直接将大型文件上传到PHP而没有使用nginx fastcgi缓冲,因此我可以立即处理来自php://input的上传数据(即作为流)。

详细说明:

我正在寻找的所需行为是让nginx在到达nginx时传递每个字节(或字节块)并将其传递到上游,在这种情况下是php-fpm。原因是能够获得非常大的文件上传并尽快处理它们(即,一旦每个块可用,就将数据移动到S3)。现在,nginx已经添加了以fastcgi_request_buffering配置选项的形式在1.7.11 http://nginx.org/en/CHANGES中禁用fastcgi缓冲的支持。

实现此功能时,似乎nginx不遵守此指令,或者php-fpm不会将数据流式传输给worker。以下是卷曲请求的结果:

[root@localhost app]# ll -h large.txt 
-rw-r--r--. 1 nginx nginx 307M Nov 10  2015 large.txt
[root@localhost app]# time curl -vs -XPOST --data-binary @large.txt http://127.0.0.1:80
* About to connect() to 127.0.0.1 port 80 (#0)
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> POST / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 127.0.0.1
> Accept: */*
> Content-Length: 321912832
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
< HTTP/1.1 404 Not Found
< Server: nginx
< Date: Fri, 06 Nov 2015 19:18:27 GMT
< Content-Type: text/html
< Content-Length: 162
< Connection: keep-alive
* HTTP error before end of send, stop sending
< 
<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Closing connection 0

real    7m43.438s
user    0m0.653s
sys 0m1.219s
[root@localhost app]# 

在这样做时,我正在拖尾nginx和php-fpm日志:

==> /var/log/php-fpm/www-error.log <==
[06-Nov-2015 12:18:27 America/Denver] PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 268173441 bytes) in Unknown on line 0

==> /var/log/nginx/error.log <==
2015/11/06 14:18:27 [error] 31657#0: *1 upstream sent unexpected FastCGI record: 3 while reading response header from upstream, client: 127.0.0.1, server: myserver.whatever.com, request: "POST / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1"
2015/11/06 14:18:27 [error] 31657#0: *1 open() "/etc/nginx/html/50x.html" failed (2: No such file or directory), client: 127.0.0.1, server: myserver.whatever.com, request: "POST / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "127.0.0.1"

==> /var/log/nginx/access.log <==
127.0.0.1 - - [06/Nov/2015:14:18:27 -0500] "POST / HTTP/1.1" 404 187 "-" "curl/7.29.0"

==> /var/log/php-fpm/error.log <==
[06-Nov-2015 14:18:27] WARNING: [pool www] child 31619 exited with code 70 after 479.683142 seconds from start
[06-Nov-2015 14:18:27] NOTICE: [pool www] child 31809 started

因此,问题很明显:PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 268173441 bytes) in Unknown on line 0

问题是,nginx是否遵守fastcgi_request_buffering off;指令,或者php-fpm流程管理器搞砸了,而不是直接将数据流式传输到php worker,而是作为整个请求对象传递。

或者,也许我错过了php-fpm的一些配置:http://php.net/manual/en/install.fpm.configuration.php

以下是一些相关的信息:

PHP fpm configs:

[root@localhost nginx]# cat /etc/php-fpm.conf | grep -vE '^;' | grep -v '^$'
include=/etc/php-fpm.d/*.conf
[global]
pid = /run/php-fpm/php-fpm.pid
error_log = /var/log/php-fpm/error.log
daemonize = yes

pool config:

[root@localhost nginx]# cat /etc/php-fpm.d/www.conf | grep -vE '^;' | grep -v '^$'
[www]
user = apache
group = apache
listen = 127.0.0.1:9000
listen.allowed_clients = 127.0.0.1
pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
slowlog = /var/log/php-fpm/www-slow.log
php_admin_value[error_log] = /var/log/php-fpm/www-error.log
php_admin_flag[log_errors] = on
php_value[session.save_handler] = files
php_value[session.save_path]    = /var/lib/php/session
php_value[soap.wsdl_cache_dir]  = /var/lib/php/wsdlcache
php_value[upload_max_filesize] = 100G
php_value[post_max_size] = 100G
php_value[max_execution_time] = 3600
request_terminate_timeout = 3600
request_slowlog_timeout = 60
php_value[max_input_time] = 3600

版本:

[root@localhost nginx]# php -v
PHP 5.6.15 (cli) (built: Oct 29 2015 14:18:11) 
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
    with Xdebug v2.3.3, Copyright (c) 2002-2015, by Derick Rethans
[root@localhost nginx]# php-fpm -v
PHP 5.6.15 (fpm-fcgi) (built: Oct 29 2015 14:18:34)
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
    with Xdebug v2.3.3, Copyright (c) 2002-2015, by Derick Rethans
[root@localhost nginx]# nginx -v
nginx version: nginx/1.8.0
[root@localhost nginx]# uname -a
Linux localhost.localdomain 3.10.0-229.14.1.el7.x86_64 #1 SMP Tue Sep 15 15:05:51 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost nginx]# cat /etc/redhat-release 
CentOS Linux release 7.1.1503 (Core)

和nginx配置:

server {
    listen       0.0.0.0:80 default_server;
    server_name               myserver.com;
    location / {
        root   /opt/my/app/public;
        index  index.html index.htm index.php;
        try_files $uri $uri/ /index.php?$args;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }

    location ~ \.php$ {
        root /opt/my/app/public;
        try_files      $uri = 404;
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
    fastcgi_request_buffering off;
    fastcgi_read_timeout 3600;
        include        fastcgi_params;
    }
}

0 个答案:

没有答案