无法获得结构<M11>的jdbc类型

时间:2019-12-23 14:43:02

标签: python jdbc pyspark apache-spark-sql aws-glue

我正在使用pyspark开发eta作业,并希望通过jdbc连接在mariadb中存储struct的数据帧。但是我得到了这个错误。 谁可以帮我?

schema = StructType(
[
    StructField('M1',
        StructType([
            StructField('M11', IntegerType(),True),
            StructField('M12', IntegerType(),True),
            StructField('M13', IntegerType(),True),   
            ])
    ),
])


df = sqlCtx.createDataFrame([],schema)

df1 = spark.createDataFrame(
[Row(
     x1=Row(field1=10, field2=1.5, x12=Row(field5='tt')), 
     x2=Row(field3="one",field4=False)
)])
df = sqlCtx.createDataFrame(df1.rdd.map(lambda x:   Row(M1=Row(M11=int(x.x1.field2), M13=None,M12=None  ))), schema)

dfx = DynamicFrame.fromDF(df, glueContext, "datasource1")

datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf(frame = dfx, catalog_connection = "db", connection_options = {"dbtable": "table1", "database": "db1"}, transformation_ctx = "datasink4")
job.commit()

0 个答案:

没有答案