本地Aws Glue Etl-使用Scala的问题

时间:2019-04-25 14:01:20

标签: scala amazon-web-services etl aws-glue glue

asJava不是Map [String,String]的成员

在创建我的端点并本地启动齐柏林飞艇之后,当我开始使用scala进行编码并执行代码时,这就是下一个错误:

import org.apache.spark.SparkContext
<console>:288: error: value asJava is not a member of Map[String,String]
               Job.init(args("JOB_NAME"), glueContext, args.asJava)

这是我的代码的样子:

%spark
import com.amazonaws.services.glue.ChoiceOption
import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.DynamicRecord
import com.amazonaws.services.glue.MappingSpec
import com.amazonaws.services.glue.ResolveSpec
import com.amazonaws.services.glue.errors.CallSite
import com.amazonaws.services.glue.util.GlueArgParser
import com.amazonaws.services.glue.util.Job
import com.amazonaws.services.glue.util.JsonOptions
import com.amazonaws.services.glue.types.StringNode
import org.apache.spark.SparkContext

object DataCleaningLambda {
    def main(sysArgs: Array[String]): Unit = {
        val spark: SparkContext = new SparkContext()
        val glueContext: GlueContext = new GlueContext(sc)
        val args = GlueArgParser.getResolvedOptions(sysArgs, Seq("JOB_NAME").toArray)
        Job.init(args("JOB_NAME"), glueContext, args.asJava)
        val datasource0 = glueContext.getCatalogSource(database = "mydatabase", tableName = "my_table", redshiftTmpDir = "", transformationContext = "datasource0").getDynamicFrame()
        val medicareDf = datasource0.toDF()
        println("here")
        Job.commit()
    } 
}

必须打印“此处”消息,但我不打印,参见图像

enter image description here

0 个答案:

没有答案