对象无法转换为流接收器

时间:2015-04-19 23:19:15

标签: scala apache-spark spark-streaming spray-json

在一个火花流应用程序中,我收到的消息是,我用来作为RDD的项类传递的可序列化类之一,无法转换为接收器。我并没有试图把它变成一个接收器,但现在是:

15/04/18 18:30:22 ERROR Executor: Exception in task 1.0 in stage 0.0 (TID 1)
java.lang.ClassCastException: vehicles.Vehicle_Status cannot be cast to org.apache.spark.streaming.receiver.Receiver
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:295)
at org.apache.spark.streaming.scheduler.ReceiverTracker$ReceiverLauncher$$anonfun$8.apply(ReceiverTracker.scala:290)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1497)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

这里的代码可能会让我在那里:

val rmqReceiver = new RMQReceiver(qAddress, "vehicle-data")
val customReceiverStream = aContext.receiverStream(rmqReceiver)

val handler = (rdd: RDD[List[String]]) => {
  this.handleStreamResult(rdd)
}
customReceiverStream.foreachRDD(handler)
def jsonToVehicleStatus(aRecord :String): Vehicle_Status ={
  val ast = aRecord.parseJson
  val aMap = ast.convertTo[Vehicle_Status]
  return aMap

}

def handleStreamResult(rdd: RDD[List[String]]): Unit ={
  rdd.foreach { list =>
    val mapList = list.map(jsonToVehicleStatus)
    val pMapList = sparkContext.parallelize(mapList)

    pMapList.saveToCassandra("vehicle_data","vehicles",AllColumns)
    println()

  }
}

有没有人知道它为什么要把它投入接收器?

0 个答案:

没有答案