这是我的代码:
val stream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet)
val lines = stream.map(_._2)
lines.print()
lines.foreachRDD { rdd =>
rdd.foreach( data =>
if (data != null) {
println(data.toString)
val records = data.toString
CassandraConnector(conf).withSessionDo {
session =>
session.execute("INSERT INTO propatterns_test.meterreadings JSON "+records+";")
}
}
)
}
所以我出错了。
我的错误日志是:
org.apache.spark.SparkException: Task not serializable
at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298)
at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
at org.apache.spark.SparkContext.clean(SparkContext.scala:2037)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:874)
at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:873)
at etc.
我正在将rdd消息存储到cassandra中。