我尝试根据JDBC To Other Databases
enter link description here将rdd保存到外部数据库,例如mysql。
dsStream.foreachRDD( rdd => {
val to_save_rdd = rdd.map(.../*some tranform operation*/)
val spark = SparkSession.builder.config(rdd.sparkContext.getConf).getOrCreate()
import spark.implicits._
to_save_rdd.toDF().show()
to_save_rdd.toDF().write
.format("jdbc")
.option("url", "jdbc:mysql://127.0.0.1:3306")
.option("dbtable", "db_name.table_name")
.option("user", "root")
.option("password", "root")
.save()
})
然而,即使我评论 to_save_rdd.toDF()。write.format(...)。option(...)。save ,它也会失败。错误日志如下。
java.lang.AssertionError: assertion failed: no symbol could be loaded from interface org.apache.hadoop.hbase.classification.InterfaceAudience$Public in object InterfaceAudience with name Public and classloader org.apache.spark.util.MutableURLClassLoader@5d20e46
at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$JavaMirror$$classToScala1(JavaMirrors.scala:1021)
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToScala$1.apply(JavaMirrors.scala:980)
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToScala$1.apply(JavaMirrors.scala:980)
at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$toScala$1.apply(JavaMirrors.scala:97)
at scala.reflect.runtime.TwoWayCaches$TwoWayCache$$anonfun$toScala$1.apply(TwoWayCaches.scala:38)
at scala.reflect.runtime.Gil$class.gilSynchronized(Gil.scala:19)
任何建议都将受到赞赏。