为什么在DStreams上运行SparkSQL会为org.apache.spark.rdd.ShuffledRDDPartition提供ClassCastException?

时间:2014-09-13 05:10:32

标签: apache-spark spark-streaming rdd apache-spark-sql

在DStream中的每个RDD上运行SparkSQL时获取ClassCastException。

java.lang.ClassCastException: org.apache.spark.rdd.ShuffledRDDPartition cannot be cast to org.apache.spark.rdd.ParallelCollectionPartition
    at org.apache.spark.rdd.ParallelCollectionRDD.compute(ParallelCollectionRDD.scala:102)
    at org.apache.spark.rdd.ParallelCollectionRDD.compute(ParallelCollectionRDD.scala:85)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
    at org.apache.spark.scheduler.Task.run(Task.scala:51)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)

0 个答案:

没有答案