我正在运行一个pyathena脚本,该脚本将分区添加到Athena外部表。数据存储在S3中。只要存在错误,就会在数据指向的S3路径中生成一个文本文件?
为什么会这样,我该如何更改?
这是我的脚本:
def partition_athena_tables(s3_dir_path, table):
s3_staging_dir = os.path.join('s3://' + s3_dir_path)
try:
# assumes you have your aws keys set in env
conn = connect(
s3_staging_dir=s3_staging_dir,
region_name='us-west-2'
)
yesterday = date.today() + timedelta(days=-1)
month = str(yesterday.month) if yesterday.month >= 10 else "0" + str(yesterday.month)
day = str(yesterday.day) if yesterday.day >= 10 else "0" + str(yesterday.day)
sql = """
ALTER TABLE db.%s
ADD PARTITION (date_col='date=2018%s%s00')
location 's3://%s/date=2018%s%s00';
"""
cursor = conn.cursor()
cursor.execute(sql % (table, month, day, s3_dir_path, month, day))
except Exception as e:
print e