Question

我正在尝试将本地目录中某个文件夹下的文件移动到s3存储桶。我知道aws sync s3命令，该命令允许将本地文件复制到s3存储桶。是否可以在go程序中执行aws sync s3命令？还有其他选择吗，也许是通过使用aws s3 sdk来完成的？

Answer 1

AWS不会通过S3公开任何sync API。我相信AWS CLI只是列出了原始文件和目标文件中的所有文件（本地文件夹和S3），计算出差异并上传/下载目标文件中缺少的所有元素。

aws sync s3的{{3}}也建议这样做：

同步目录和S3前缀。将新文件和更新文件从源目录递归复制到目标。如果它们包含一个或多个文件，则仅在目标位置创建文件夹。

您将能够使用AWS SDK列出远程对象，并将对象从本地对象上传到S3，但是您必须管理本地列表以及自己的代码中的上传机制。

我没有使用过AWS开发工具包的第一手经验，但是在documentation中进行的粗略搜索不会产生与sync类似的结果。我可以肯定地说，尽管没有用于Python和Javascript的官方SDK公开任何sync方法。

Answer 2

您需要一次上传文件。

以下是Performing Basic Amazon S3 Bucket Operations - AWS SDK for Go中的一个示例：

/*
   Copyright 2010-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.

   This file is licensed under the Apache License, Version 2.0 (the "License").
   You may not use this file except in compliance with the License. A copy of
   the License is located at

    http://aws.amazon.com/apache2.0/

   This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
   CONDITIONS OF ANY KIND, either express or implied. See the License for the
   specific language governing permissions and limitations under the License.
*/

package main

import (
    "github.com/aws/aws-sdk-go/aws"
    "github.com/aws/aws-sdk-go/aws/session"
    "github.com/aws/aws-sdk-go/service/s3/s3manager"
    "fmt"
    "os"
)

// Creates a S3 Bucket in the region configured in the shared config
// or AWS_REGION environment variable.
//
// Usage:
//    go run s3_upload_object.go BUCKET_NAME FILENAME
func main() {
    if len(os.Args) != 3 {
        exitErrorf("bucket and file name required\nUsage: %s bucket_name filename",
            os.Args[0])
    }

    bucket := os.Args[1]
    filename := os.Args[2]

    file, err := os.Open(filename)
    if err != nil {
        exitErrorf("Unable to open file %q, %v", err)
    }

    defer file.Close()

    // Initialize a session in us-west-2 that the SDK will use to load
    // credentials from the shared credentials file ~/.aws/credentials.
    sess, err := session.NewSession(&aws.Config{
        Region: aws.String("us-west-2")},
    )

    // Setup the S3 Upload Manager. Also see the SDK doc for the Upload Manager
    // for more information on configuring part size, and concurrency.
    //
    // http://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/#NewUploader
    uploader := s3manager.NewUploader(sess)

    // Upload the file's body to S3 bucket as an object with the key being the
    // same as the filename.
    _, err = uploader.Upload(&s3manager.UploadInput{
        Bucket: aws.String(bucket),

        // Can also use the `filepath` standard library package to modify the
        // filename as need for an S3 object key. Such as turning absolute path
        // to a relative path.
        Key: aws.String(filename),

        // The file to be uploaded. io.ReadSeeker is preferred as the Uploader
        // will be able to optimize memory when uploading large content. io.Reader
        // is supported, but will require buffering of the reader's bytes for
        // each part.
        Body: file,
    })
    if err != nil {
        // Print the error and exit.
        exitErrorf("Unable to upload %q to %q, %v", filename, bucket, err)
    }

    fmt.Printf("Successfully uploaded %q to %q\n", filename, bucket)
}

func exitErrorf(msg string, args ...interface{}) {
    fmt.Fprintf(os.Stderr, msg+"\n", args...)
    os.Exit(1)
}

将文件从本地文件夹移动到Go中的s3存储桶

2 个答案: