我正在使用S3兼容的后端,它不支持MultipartUpload。
我有一个疏远的情况,其中一些服务器上传文件时,它完成正常但在其他服务器boto3自动尝试使用MultipartUpload上传文件。我正在尝试上传的文件完全相同,用于测试相同的后端,区域/租户,存储桶等...
如documentation所示,MultipartUpload会在需要时自动启用:
- 当文件超过特定大小阈值时自动切换到多部分传输
以下是自动切换到MultipartUpload时的一些日志:
自动切换到MultipartUpload时记录:
DEBUG:botocore.hooks:Event request-created.s3.CreateMultipartUpload: calling handler <function enable_upload_callbacks at 0x2b001b8>
DEBUG:botocore.endpoint:Sending http request: <PreparedRequest [POST]>
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mytenant.mys3backend.cloud.corp
DEBUG:botocore.vendored.requests.packages.urllib3.connectionpool:"POST /cassandra/samplefile.tgz?uploads HTTP/1.1" 501 None
DEBUG:botocore.parsers:Response headers: {'date': 'Fri, 18 Dec 2015 09:12:48 GMT', 'transfer-encoding': 'chunked', 'content-type': 'application/xml;charset=UTF-8', 'server': 'HCP V7.2.0.26'}
DEBUG:botocore.parsers:Response body:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>NotImplemented</Code>
<Message>The request requires functionality that is not implemented in the current release</Message>
<RequestId>1450429968948</RequestId>
<HostId>aGRpLmJvc3RoY3AuY2xvdWQuY29ycDoyNg==</HostId>
</Error>
DEBUG:botocore.hooks:Event needs-retry.s3.CreateMultipartUpload: calling handler <botocore.retryhandler.RetryHandler object at 0x2a490d0>
不从其他服务器切换到multipart的日志,但是对于同一文件:
DEBUG:botocore.hooks:Event request-created.s3.PutObject: calling handler <function enable_upload_callbacks at 0x7f436c025500>
DEBUG:botocore.endpoint:Sending http request: <PreparedRequest [PUT]>
INFO:botocore.vendored.requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mytenant.mys3backend.cloud.corp
DEBUG:botocore.awsrequest:Waiting for 100 Continue response.
DEBUG:botocore.awsrequest:100 Continue response seen, now sending request body.
DEBUG:botocore.vendored.requests.packages.urllib3.connectionpool:"PUT /cassandra/samplefile.tgz HTTP/1.1" 200 0
DEBUG:botocore.parsers:Response headers: {'date': 'Fri, 18 Dec 2015 10:05:25 GMT', 'content-length': '0', 'etag': '"b407e71de028fe62fd9f2f799e606855"', 'server': 'HCP V7.2.0.26'}
DEBUG:botocore.parsers:Response body:
DEBUG:botocore.hooks:Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7f436be1ecd0>
DEBUG:botocore.retryhandler:No retry needed.
我上传的文件如下:
connection = boto3.client(service_name='s3',
region_name='',
api_version=None,
use_ssl=True,
verify=True,
endpoint_url=url,
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
aws_session_token=None,
config=None)
connection.upload_file('/tmp/samplefile.tgz','mybucket','remotefile.tgz')
问题是:
答案 0 :(得分:4)
当我看到boto3时,遇到了你的问题
当文件超过a时自动切换到多部分传输 特定大小阈值??
是upload_file(来自client / resource / S3Transfer)将自动转换为分段上传,默认阈值大小为8 MB。
如果您不想使用MultiPart,那么永远不要使用upload_file方法,只使用put_object方法,而不是Multipart。
client = boto3.client(&#39; s3&#39;)
client.put_object(体=开放(&#39; /test.csv'),铲斗=&#39; mybucket&#39;,答案=&#39; test.csv&#39)