使用Flask处理超大文件上传（1 GB +）的最佳方法是什么？
我的应用程序本质上需要多个文件为它们分配一个唯一的文件编号，然后根据用户选择的位置将其保存在服务器上。
我们如何将文件上传作为后台任务运行，以便用户不会让浏览器旋转1小时，而是可以立即进入下一页？
答案 0 :(得分：6)
我认为解决该问题的超级简单方法只是将文件分批发送。因此，要完成这项工作将需要两个部分，即前端（网站）和后端（服务器）。
对于前端部分，您可以使用Dropzone.js之类的东西，它没有附加的依赖关系，还包括不错的CSS。您所要做的就是将dropzone类添加到表单中，它会自动将其变成其特殊的拖放字段之一（您也可以单击并选择）。
但是，默认情况下，dropzone不会对文件进行分块。幸运的是，它确实很容易启用。
这是启用了DropzoneJS和chunking的示例文件上传表单：
<html lang="en">
<head>
<meta charset="UTF-8">
<link rel="stylesheet"
href="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.css"/>
<link rel="stylesheet"
href="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/basic.min.css"/>
<script type="application/javascript"
src="https://cdnjs.cloudflare.com/ajax/libs/dropzone/5.4.0/min/dropzone.min.js">
</script>
<title>File Dropper</title>
</head>
<body>
<form method="POST" action='/upload' class="dropzone dz-clickable"
id="dropper" enctype="multipart/form-data">
</form>
<script type="application/javascript">
Dropzone.options.dropper = {
paramName: 'file',
chunking: true,
forceChunking: true,
url: '/upload',
maxFilesize: 1025, // megabytes
chunkSize: 1000000 // bytes
}
</script>
</body>
</html>
这是使用flask的后端部分：
import logging
import os
from flask import render_template, Blueprint, request, make_response
from werkzeug.utils import secure_filename
from pydrop.config import config
blueprint = Blueprint('templated', __name__, template_folder='templates')
log = logging.getLogger('pydrop')
@blueprint.route('/')
@blueprint.route('/index')
def index():
# Route to serve the upload form
return render_template('index.html',
page_name='Main',
project_name="pydrop")
@blueprint.route('/upload', methods=['POST'])
def upload():
file = request.files['file']
save_path = os.path.join(config.data_dir, secure_filename(file.filename))
current_chunk = int(request.form['dzchunkindex'])
# If the file already exists it's ok if we are appending to it,
# but not if it's new file that would overwrite the existing one
if os.path.exists(save_path) and current_chunk == 0:
# 400 and 500s will tell dropzone that an error occurred and show an error
return make_response(('File already exists', 400))
try:
with open(save_path, 'ab') as f:
f.seek(int(request.form['dzchunkbyteoffset']))
f.write(file.stream.read())
except OSError:
# log.exception will include the traceback so we can see what's wrong
log.exception('Could not write to file')
return make_response(("Not sure why,"
" but we couldn't write the file to disk", 500))
total_chunks = int(request.form['dztotalchunkcount'])
if current_chunk + 1 == total_chunks:
# This was the last chunk, the file should be complete and the size we expect
if os.path.getsize(save_path) != int(request.form['dztotalfilesize']):
log.error(f"File {file.filename} was completed, "
f"but has a size mismatch."
f"Was {os.path.getsize(save_path)} but we"
f" expected {request.form['dztotalfilesize']} ")
return make_response(('Size mismatch', 500))
else:
log.info(f'File {file.filename} has been uploaded successfully')
else:
log.debug(f'Chunk {current_chunk + 1} of {total_chunks} '
f'for file {file.filename} complete')
return make_response(("Chunk upload successful", 200))
答案 1 :(得分：1)
使用copy_current_request_context，它将复制上下文request。因此，您可以使用线程或其他任何方式使任务在后台运行。
也许可以清楚地说明一个例子。我已经通过3.37G文件-debian-9.5.0-amd64-DVD-1.iso对它进行了测试。
# coding:utf-8
from flask import Flask,render_template,request,redirect,url_for
from werkzeug.utils import secure_filename
import os
from time import sleep
from flask import copy_current_request_context
import threading
import datetime
app = Flask(__name__)
@app.route('/upload', methods=['POST','GET'])
def upload():
@copy_current_request_context
def save_file(closeAfterWrite):
print(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S') + " i am doing")
f = request.files['file']
basepath = os.path.dirname(__file__)
upload_path = os.path.join(basepath, '',secure_filename(f.filename))
f.save(upload_path)
closeAfterWrite()
print(datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S') + " write done")
def passExit():
pass
if request.method == 'POST':
f= request.files['file']
normalExit = f.stream.close
f.stream.close = passExit
t = threading.Thread(target=save_file,args=(normalExit,))
t.start()
return redirect(url_for('upload'))
return render_template('upload.html')
if __name__ == '__main__':
app.run(debug=True)
这是临时的，应该是templates \ upload.html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Title</title>
</head>
<body>
<h1>example</h1>
<form action="" enctype='multipart/form-data' method='POST'>
<input type="file" name="file">
<input type="submit" value="upload">
</form>
</body>
</html>
答案 2 :(得分：0)
上传文件时，您无法离开页面并继续播放。该页面必须保持打开状态才能继续上传。
您可以做的是打开一个新选项卡，仅用于处理上载，并在用户无意中关闭新选项卡而在完成上传之前提醒用户。这样，上传将与用户在原始页面上执行的操作分开，因此他们仍然可以导航而无需取消上传。上传标签也可以在完成后自行关闭。
index.js
// get value from <input id="upload" type="file"> on page
var upload = document.getElementById('upload');
upload.addEventListener('input', function () {
// open new tab and stick the selected file in it
var file = upload.files[0];
var uploadTab = window.open('/upload-page', '_blank');
if (uploadTab) {
uploadTab.file = file;
} else {
alert('Failed to open new tab');
}
});
upload-page.js
window.addEventListener('beforeunload', function () {
return 'The upload will cancel if you leave the page, continue?';
});
window.addEventListener('load', function () {
var req = new XMLHttpRequest();
req.addEventListener('progress', function (evt) {
var percentage = '' + (evt.loaded / evt.total * 100) + '%';
// use percentage to update progress bar or something
});
req.addEventListener('load', function () {
alert('Upload Finished');
window.removeEventListener('beforeunload');
window.close();
});
req.addRequestHeader('Content-Type', 'application/octet-stream');
req.open('POST', '/upload/'+encodeURIComponent(window.file.name));
req.send(window.file);
});
在服务器上，您可以使用request.stream读取大块上载的文件，以避免必须等待整个内容首先加载到内存中。
server.py
@app('/upload/<filename>', methods=['POST'])
def upload(filename):
filename = urllib.parse.unquote(filename)
bytes_left = int(request.headers.get('content-length'))
with open(os.path.join('uploads', filename), 'wb') as upload:
chunk_size = 5120
while bytes_left > 0:
chunk = request.stream.read(chunk_size)
upload.write(chunk)
bytes_left -= len(chunk)
return make_response('Upload Complete', 200)
您也许可以使用FormData api而不是八位字节流，但是我不确定是否可以在flask中进行流式传输。