Question

我有3个水桶1.commonfolder 2.jsonfolder 3.csvfolder。

公用文件夹将同时包含json和csv文件
需要将所有csv文件复制到csvfolder
需要将所有json文件复制到json文件夹

下面的代码可用于获取commonfolder之后的所有文件

import boto3

s3 = boto3.client('s3')
def lambda_handler(event, context):
    #List all the bucket names
    response = s3.list_buckets()
    for bucket in response['Buckets']:
        print (bucket)
        print(f'{bucket["Name"]}')
        #Get the files of particular bucket
        if bucket["Name"] == 'tests3json':
         
            resp = s3.list_objects_v2(Bucket='commonfolder')
            for obj in resp['Contents']:
                files = obj['Key']
                print(files)

            if(filename.split('.')[1].lower()=='json'):
                copyjson(bucket,filename)
                #copyjson(jsonfolder,filename)
            elif(filename.split('.')[1].lower()=='csv'):
                copycsv(bucket, filename)
                #copycsv(csvfolder,filename)

需要创建一个新函数copyjson，copycsv来完成这项工作
需要根据文件扩展名从common-bucket复制到csv-bucket或json-bucket

Answer 1

您可以检查以下代码：

import boto3

s3 = boto3.resource('s3')

def lambda_handler(event, context):
    
    source_bucket = s3.Bucket('01-commonfolder-231')
    json_bucket = s3.Bucket('02-jsonfolder-3435')
    csv_bucket = s3.Bucket('03-csvfolder-4552')
    
    for object in source_bucket.objects.all():
        
        #print(object)
        
        if object.key.endswith('.json'):
            
            print(f"{object.key} to json bucket")
      
            copy_object = json_bucket.Object(object.key)
            copy_object.copy({'Bucket': object.bucket_name,
                              'Key': object.key})                             
            
        elif object.key.endswith('.csv'):
            
            print(f"{object.key} to csv bucket")            

            copy_object = csv_bucket.Object(object.key)
            copy_object.copy({'Bucket': object.bucket_name,
                              'Key': object.key})

我使用自己的带有测试文件的样本存储桶对此进行了测试：

aaa.json to json bucket
bbbbb.csv to csv bucket
bbbbb.json to json bucket
hhhh.csv to csv bucket

Answer 2

您可以使用move()中的shutil方法：

from shutil import move
from glob import glob

common_folder = 'C:\\Users\\User\\Desktop\\commonfolder\\'
csv_folder = 'C:\\Users\\User\\Desktop\\csvfolder\\'
json_folder = 'C:\\Users\\User\\Desktop\\jsonfolder\\'

for csv in glob(common_folder+"*.csv"):
    move(csv, csv_folder)

for json in glob(common_folder+"*.json"):
    move(json, json_folder)

如何在特定后缀s3中从一个存储桶复制到另一个存储桶

2 个答案: