java.lang.NoClassDefFoundError:org / apache / spark / sql / execution / datasources / jdbc / JDBCOptions

时间:2018-08-16 08:57:00

标签: scala apache-spark jdbc hive

我正在尝试将数据加载到hive表中,而我却遇到以下错误

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/execution/datasources/jdbc/JDBCOptions$
    at com.rxcorp.canada_taunus.LoadData.loadDFToDB(LoadData.scala:141)
    at com.rxcorp.canada_taunus.CanadaTaunus$.main(xxxxxxx.scala:149)
    at com.rxcorp.canada_taunus.CanadaTaunus.main(xxxxxxx.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions$
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 3 more

代码

import org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions

    val jdbcOptions = new org.apache.spark.sql.execution.datasources.jdbc.JDBCOptions(Map(
            JDBCOptions.JDBC_URL->" jdbc:hive2://xxxxx.xxxxx.com:10000/dev_batch;principal=hive/xxxxx.xxxxx.com@internal.xxxglobal.com;ssl=1;mapred.job.queue.name=cl;AuthMech=3;username=xxxxx;password=xxxx",
            JDBCOptions.JDBC_TABLE_NAME-> "testTable",
            JDBCOptions.JDBC_BATCH_INSERT_SIZE->"10000",
            JDBCOptions.JDBC_DRIVER_CLASS -> "com.cloudera.hive.jdbc41.HS2Driver",
            "user"->"xxxxx",
            "password"->"xxxxxx",
            "SSLTrustStore"-> "jklecacerts"

          ))

      org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils.saveTable(dataFrame,Option(dataFrame.schema),false.asInstanceOf[Boolean],jdbcOptions)

0 个答案:

没有答案