Spark根据jdbc将数据保存到MYSQL到其他数据库

时间:2017-10-19 07:34:21

标签: jdbc spark-dataframe spark-streaming

我尝试根据JDBC To Other Databases enter link description here将rdd保存到外部数据库,例如mysql。

dsStream.foreachRDD( rdd => {
  val to_save_rdd = rdd.map(.../*some tranform operation*/)    
  val spark = SparkSession.builder.config(rdd.sparkContext.getConf).getOrCreate()
  import spark.implicits._
  to_save_rdd.toDF().show()
  to_save_rdd.toDF().write
  .format("jdbc")
  .option("url", "jdbc:mysql://127.0.0.1:3306")
  .option("dbtable", "db_name.table_name")
  .option("user", "root")
  .option("password", "root")
  .save()

})

然而,即使我评论 to_save_rdd.toDF()。write.format(...)。option(...)。save ,它也会失败。错误日志如下。

java.lang.AssertionError: assertion failed: no symbol could be loaded from interface org.apache.hadoop.hbase.classification.InterfaceAudience$Public in object InterfaceAudience with name Public and classloader org.apache.spark.util.MutableURLClassLoader@5d20e46
        at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$JavaMirror$$classToScala1(JavaMirrors.scala:1021)
            at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToScala$1.apply(JavaMirrors.scala:980)
            at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToScala$1.apply(JavaMirrors.scala:980)
            at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$toScala$1.apply(JavaMirrors.scala:97)
            at scala.reflect.runtime.TwoWayCaches$TwoWayCache$$anonfun$toScala$1.apply(TwoWayCaches.scala:38)
            at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)

任何建议都将受到赞赏。

0 个答案:

没有答案