我已经在堆栈溢出中进行了搜索,以了解如何以源文件名作为值附加新列。 但是,它没有按预期进行。
在最终的实木复合地板文件中,我找到了一个名为input_file_name的新列,但该值为空。 (如“”)
我想知道我忽略了哪一步。
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql import functions as F
from awsglue.dynamicframe import DynamicFrame
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "mydb", table_name = "mytable", transformation_ctx = "datasource0")
datasource1 = datasource0.toDF().withColumn("input_file_name", F.input_file_name())
datasource2 = DynamicFrame.fromDF(datasource1, glueContext, "datasource2")
applymapping1 = ApplyMapping.apply(frame = datasource2, mappings = [("input_file_name", "string", "input_file_name", "string"),
("Profile", "struct", "Profile","struct")], transformation_ctx = "applymapping1")
datasink4 = glueContext.write_dynamic_frame.from_options(frame = applymapping1, connection_type = "s3", connection_options = {"path": "s3://temp/testing"}, format = "parquet", transformation_ctx = "datasink4")
job.commit()
答案 0 :(得分:0)
您不能通过将动态框架转换为数据框架来使用input_file_name()
您必须使用spark.read api将数据读取到数据框。下一个直接步骤必须是使用input_file_name()-在对数据帧执行任何操作之前。
请记住,aws胶水已知可处理高达45gb数据,然后引发错误。增加dpu无效