使用boto3

Question

下面的代码使用本地计算机上的Pycham从AWS s3读取CSV文件。

# Read CSV from s3

import os
import boto3
import pandas as pd
import sys

if sys.version_info[0] < 3:
    from StringIO import StringIO  # Python 2.x
else:
    from io import StringIO

aws_id = 'XXXXXXXXXXXXXXX'
aws_secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

client = boto3.client('s3', aws_access_key_id=aws_id, aws_secret_access_key=aws_secret)

bucket_name = 'bucket-name'
object_key = 'folder-name/test.csv'

csv_obj = client.get_object(Bucket=bucket_name, Key=object_key)

body = csv_obj['Body']

csv_string = body.read().decode('utf-8')

df = pd.read_csv(StringIO(csv_string))

x = df.head()

print(x)

我希望能够以相同的方式读取多个CSV文件。文件夹中几乎所有内容。

我的文件在以下目录中：

bucket-name/folder-name/year=2018/month=01/file_032342.csv
bucket-name/folder-name/year=2018/month=02/file_434423.csv
bucket-name/folder-name/year=2018/month=03/file_343254.csv
bucket-name/folder-name/year=2018/month=04/file_544353.csv