如何使用Boto3在S3的文件夹中进行迭代?

时间:2018-12-18 22:27:36

标签: python django amazon-s3 amazon-ec2 boto3

在s3的实例中,我有一个包含N个文件的文件夹,我需要在下面使用此脚本进行迭代,我需要获取所有文件并将其转换,该脚本托管在使用django的ec2实例上。

我通过使用boto3函数get_object进行了很多尝试,但我得到的只是什么。

有人可以告诉我我该怎么做吗?我需要在转换之前下载此文件,还是可以直接这样做?

 def upload_folder_to_s3(local_folder, destination_folder, s3_bucket):
    '''
    Function to upload a specific local folder to S3 bucket.

    Parameters:
    local_folder (str): Path to local folder.
    destination_folder (str): Path to destination folder on S3.
    s3_bucket (str): Bucket name on S3.

    Return:
    '''
    # Global variables
    global client

    # Iterate over files on folder
    for root, dirs, files in os.walk(local_folder):
        for filename in files:
            print(filename)
            # construct the full local path
            local_path = os.path.join(root, filename)

            # construct the full Dropbox path
            relative_path = os.path.relpath(local_path, local_folder)
            s3_path = os.path.join(destination_folder, relative_path)

            # relative_path = os.path.relpath(os.path.join(root, filename))

            print('Searching "%s" in "%s"' % (s3_path, s3_bucket))
            try:
                client.get_object(Bucket=s3_bucket, Key=s3_path)
                print("Path found on S3! Skipping %s..." % s3_path)

                # try:
                    # client.delete_object(Bucket=bucket, Key=s3_path)
                # except:
                    # print "Unable to delete %s..." % s3_path
            except:
                print("Uploading %s..." % s3_path)
                client.upload_file(local_path, s3_bucket, s3_path)
    return local_folder

0 个答案:

没有答案