pyathena脚本是否将日志存储在它指向的s3存储桶中?

时间:2018-07-12 07:59:13

标签: amazon-s3 amazon-athena pyathena

我正在运行一个pyathena脚本,该脚本将分区添加到Athena外部表。数据存储在S3中。只要存在错误,就会在数据指向的S3路径中生成一个文本文件?

为什么会这样,我该如何更改?

这是我的脚本:

def partition_athena_tables(s3_dir_path, table):
    s3_staging_dir = os.path.join('s3://' + s3_dir_path)
    try:
        # assumes you have your aws keys set in env
        conn = connect(
            s3_staging_dir=s3_staging_dir,
            region_name='us-west-2'
        )

        yesterday = date.today() + timedelta(days=-1)
        month = str(yesterday.month) if yesterday.month >= 10 else "0" + str(yesterday.month)
        day = str(yesterday.day) if yesterday.day >= 10 else "0" + str(yesterday.day)
        sql = """
                 ALTER TABLE db.%s 
                 ADD PARTITION (date_col='date=2018%s%s00') 
                 location 's3://%s/date=2018%s%s00'; 
            """
        cursor = conn.cursor()
        cursor.execute(sql % (table, month, day, s3_dir_path, month, day))

    except Exception as e:
        print e

0 个答案:

没有答案