如何在Scala中将DataFrame投射到DynamicFrame

时间:2018-09-07 08:42:51

标签: scala amazon-web-services apache-spark apache-spark-sql aws-glue

val ruleDs = glueContext.getSource(connectionType = "dynamodb",connectionOptions= JsonOptions(Map("dynamodb.input.tableName" -> "rules"))).getDynamicFrame()
ruleDs.toDF().createOrReplaceTempView("rules")
val ruleSqlDf = glueContext.getSparkSession.sql(s"Select * from rules where id='$ruleID'")
val ruleSqlDyf = DynamicFrame(ruleSqlDf, glueContext).withName("ruleSqlDyf")
ruleDs.printSchema()
/*result 
|-- Conditions: string
|-- InstalledApps: struct
| |-- applicationPackages: set
| | |-- element: string */

ruleSqlDyf.printSchema()
/*result
|-- Conditions: null
|-- InstalledApps: struct
| |-- applicationPackages: array
| | |-- element: string */

如何将DataFrame投射到DynamicFrame?当我通过'ruleDs'创建新的DynamicFrame时,架构会发生变化

0 个答案:

没有答案