Question

我使用Spark / Scala创建了一个对象，用于将数据从Oracle源加载到Hive Table。数据库密码通过typesafe.ConfigFactory通过application.properties传递。我尝试在用户文件夹以及其他尝试的CLASSPATH中使用APPLICATION.CONF，并使用以下spark-submit。

每次尝试时，我都遇到错误，提示“ java.lang.IllegalArgumentException：实例化'org.apache.spark.sql.hive.HiveSessionStateBuilder'时出错：”。表示类似属性未达到configFactory方法有人可以帮助我吗？

//my object snippet
object LoadFromOracleToHive {
  def SaveToHive(spark :SparkSession):Unit = {
    try {
      val appConf = ConfigFactory.load(s"application.conf").getConfig("my.config")
      val sparkConfig = appConf.getConfig("spark")  //  config.getConfig("spark")
      val df = spark
        .read
        .format("jdbc")
        .options(Map("password" -> sparkConfig.getString("password")  , "driver" -> "oracle.jdbc.driver.OracleDriver"))
//my application.conf       
my.config {
  spark {
    password = "password"
  }
}
//my spark-submit
spark-submit --class LoadFromOracleToHive \
--master yarn \
--deploy-mode client \
--driver-memory 4g \
--executor-memory 8g \
--num-executors 15 \
--executor-cores 5 \
--conf spark.kryoserializer.buffer.max=512m  \
--queue csg \
--jars /home/myuserfolder/ojdbc7.jar  /home/myuserfolder/SandeepTest-1.0-SNAPSHOT-jar-with-dependencies.jar \
--queue /home/myuserfolder/application.conf \
--conf spark.driver.extraClassPath=-Dconfig.file=/home/myuserfolder/application.conf \
--conf spark.executor.extraClassPath=-Dconfig.file=/home/myuserfolder/application.conf

将应用程序配置参数传递给Spark应用程序时出现问题

0 个答案: