Question

我想在S3上保存长时间运行的作业的结果。这项工作是用Python实现的，所以我使用的是boto3。 user guide表示使用S3.Client.upload_fileobj来实现此目的，但我无法弄清楚如何检查上传是否成功。根据文档，该方法不会返回任何内容，也不会引发错误。 Callback param似乎用于进度跟踪而不是错误检查。还不清楚方法调用是同步还是异步。

如果上传因任何原因失败，我想将内容保存到磁盘并记录错误。所以我的问题是：如何检查boto3 S3.Client.upload_fileobj调用是否成功，如果失败则进行一些错误处理？

Answer 1

我建议您执行以下操作 -

  try:
     response = upload_fileobj()
  except Exception as e:
     save the contents to the disk and log an error.
  if response is None:  

     polling after every 10s to check if the file uploaded successfully or not using **head_object()** function..
     If you got the success response from head_object :
          break
     If you got error in accessing the object:
        save the contents to the disk and log an error.

所以，基本上使用head_object()进行民意调查

Answer 2

我使用head_object和wait_until_exists的组合。

import boto3

from botocore.exceptions import ClientError, WaiterError

session = boto3.Session()
s3_client = session.client('s3')
s3_resource = session.resource('s3')


def upload_src(src, filename, bucketName):
    success = False

    try:
        bucket = s3_resource.Bucket(bucketName)
    except ClientError as e:
        bucket = None

    try:
        # In case filename already exists, get current etag to check if the 
        # contents change after upload
        head = s3_client.head_object(Bucket=bucketName, Key=filename)
    except ClientError:
        etag = ''
    else:
        etag = head['ETag'].strip('"')

    try:
        s3_obj = bucket.Object(filename)
    except ClientError, AttributeError:
        s3_obj = None

    try:
        s3_obj.upload_fileobj(src)
    except ClientError, AttributeError:
        pass
    else:
        try:
            s3_obj.wait_until_exists(IfNoneMatch=etag)
        except WaiterError as e:
            pass
        else:
            head = s3_client.head_object(Bucket=bucketName, Key=filename)
            success = head['ContentLength']

    return success

Answer 3

boto3.resource对象中似乎有一个wait_until_exists() helper function用于此目的

这是我们的使用方式： s3_client.upload_fileobj（文件，BUCKET_NAME，文件路径） s3_resource.Object（BUCKET_NAME，file_path）.wait_until_exists（）

如何检查boto3 S3.Client.upload_fileobj是否成功？

3 个答案: