Question

我需要在AWS S3中重命名一大堆对象。对于小型对象，以下代码段完美无缺：

input := &s3.CopyObjectInput{
    Bucket:     aws.String(bucket),
    Key:        aws.String(targetPrefix),
    CopySource: aws.String(source),
}
_, err = svc.CopyObject(input)
if err != nil {
    panic(errors.Wrap(err, "error copying object"))
}

我遇到了较大对象的S3大小限制。我知道我需要使用多部分上传来复制对象。这是我到目前为止所尝试的：

multiPartUpload, err := svc.CreateMultipartUpload(
    &s3.CreateMultipartUploadInput{
        Bucket: aws.String(bucket),
        Key: aws.String(targetPrefix), // targetPrefix is the new name
    },
)
if err != nil {
    panic(errors.Wrap(err, "could not create MultiPartUpload"))
}
resp, err := svc.UploadPartCopy(
    &s3.UploadPartCopyInput{
        UploadId:   multiPartUpload.UploadId,
        Bucket:     aws.String(bucket),
        Key:        aws.String(targetPrefix),
        CopySource: aws.String(source),
        PartNumber: aws.Int64(1),
    },
)
if err != nil {
    panic(errors.Wrap(err, "error copying multipart object"))
}
log.Printf("copied: %v", resp)

Golang SDK拯救了我：

InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120

我也尝试了以下方法，但我没有在此列出任何部分：

multiPartUpload, err := svc.CreateMultipartUpload(
    &s3.CreateMultipartUploadInput{
        Bucket: aws.String(bucket),
        Key: aws.String(targetPrefix), // targetPrefix is the new name
    },
)
if err != nil {
    panic(errors.Wrap(err, "could not create MultiPartUpload"))
}
err = svc.ListPartsPages(
    &s3.ListPartsInput{
        Bucket:   aws.String(bucket),       // Required
        Key:      obj.Key,                  // Required
        UploadId: multiPartUpload.UploadId, // Required
    },
    // Iterate over all parts in the `CopySource` object
    func(parts *s3.ListPartsOutput, lastPage bool) bool {
        log.Printf("parts:\n%v\n%v", parts, parts.Parts)
        // parts.Parts is an empty slice
        for _, part := range parts.Parts {
            log.Printf("copying %v part %v", source, part.PartNumber)
            resp, err := svc.UploadPartCopy(
                &s3.UploadPartCopyInput{
                    UploadId:   multiPartUpload.UploadId,
                    Bucket:     aws.String(bucket),
                    Key:        aws.String(targetPrefix),
                    CopySource: aws.String(source),
                    PartNumber: part.PartNumber,
                },
            )
            if err != nil {
                panic(errors.Wrap(err, "error copying object"))
            }
            log.Printf("copied: %v", resp)
        }
        return true
    },
)
if err != nil {
    panic(errors.Wrap(err, "something went wrong with ListPartsPages!"))
}

我做错了什么或者我错过了什么？

Answer 1

我认为ListPartsPages是错误的方向，因为它适用于＆＃34; Multipart上传＆＃34;这是一个与s3＆＃34; Object＆＃34;不同的实体。因此，您已将已上传的部分列入刚刚创建的分段上传。

您的第一个示例已接近所需，但您需要手动将原始文件拆分为多个部分，每个部分的范围由UploadPartCopyInput＆＃39; s {{1}指定}。至少这是我从阅读文档中获得的。

使用MultiPartUpload复制S3对象

1 个答案: