org.apache.spark.SparkException:任务不可序列化?

时间:2016-10-13 06:44:04

标签: apache-spark cassandra streaming

这是我的代码:

val stream = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet)

      val lines = stream.map(_._2)
      lines.print()
      lines.foreachRDD { rdd =>
        rdd.foreach( data =>
          if (data != null) {
            println(data.toString)
            val records = data.toString
            CassandraConnector(conf).withSessionDo {
              session =>
                session.execute("INSERT INTO propatterns_test.meterreadings JSON "+records+";")
            }
          }
        )
      }

所以我出错了。

我的错误日志是:

org.apache.spark.SparkException: Task not serializable
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:298)
    at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
    at org.apache.spark.SparkContext.clean(SparkContext.scala:2037)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:874)
    at org.apache.spark.rdd.RDD$$anonfun$foreach$1.apply(RDD.scala:873)
    at etc.

我正在将rdd消息存储到cassandra中。

0 个答案:

没有答案
相关问题