Spark Streaming中的java.lang.NullPointerException

Question

我有一个Spark / Kafka流媒体应用程序。该应用程序将与oracle数据库交互以存储偏移并保存一些数据。

成功播放数小时后，应用失败并出现以下错误。

我能否知道Spark-Kafka流媒体应用程序中可能导致这些错误的原因。

这是我从日志中看到的......

Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

2017/01/05 21:14:00 ERROR JobScheduler: Error in job generator
java.lang.NullPointerException
        at org.apache.spark.storage.BlockManagerMaster.removeRdd(BlockManagerMaster.scala:104)
        at org.apache.spark.SparkContext.unpersistRDD(SparkContext.scala:1612)
        at org.apache.spark.rdd.RDD.unpersist(RDD.scala:203)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$3.apply(DStream.scala:469)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$3.apply(DStream.scala:468)
        at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:107)
        at scala.collection.mutable.HashMap$$anon$2$$anonfun$foreach$3.apply(HashMap.scala:107)
        at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
        at scala.collection.mutable.HashMap$$anon$2.foreach(HashMap.scala:107)
        at org.apache.spark.streaming.dstream.DStream.clearMetadata(DStream.scala:468)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$5.apply(DStream.scala:481)
        at org.apache.spark.streaming.dstream.DStream$$anonfun$clearMetadata$5.apply(DStream.scala:481)
        at scala.collection.immutable.List.foreach(List.scala:318)
        at org.apache.spark.streaming.dstream.DStream.clearMetadata(DStream.scala:481)
        at org.apache.spark.streaming.DStreamGraph$$anonfun$clearMetadata$2.apply(DStreamGraph.scala:127)
        at org.apache.spark.streaming.DStreamGraph$$anonfun$clearMetadata$2.apply(DStreamGraph.scala:127)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at org.apache.spark.streaming.DStreamGraph.clearMetadata(DStreamGraph.scala:127)
        at org.apache.spark.streaming.scheduler.JobGenerator.clearMetadata(JobGenerator.scala:261)
        at org.apache.spark.streaming.scheduler.JobGenerator.org$apache$spark$streaming$scheduler$JobGenerator$$processEvent(JobGenerator.scala:182)
        at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:87)
        at org.apache.spark.streaming.scheduler.JobGenerator$$anon$1.onReceive(JobGenerator.scala:86)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

驱动程序日志报告这些错误

    2017/01/06 16:34:03 ERROR ContextCleaner: Error cleaning broadcast 18
    2017/01/06 16:34:13 ERROR ContextCleaner: Error cleaning broadcast 22
    2017/01/06 16:34:23 ERROR ContextCleaner: Error cleaning broadcast 20

Master log中没什么奇怪的。

17/01/06 17:05:38 INFO Master: Removing executor app-20170105220118-0005/34944 because it is EXITED
17/01/06 17:05:38 INFO Master: Launching executor app-20170105220118-0005/34947 on worker worker-20170105215345-10.251.228.107-53462
17/01/06 17:05:43 INFO Master: Removing executor app-20170105220118-0005/34945 because it is EXITED
17/01/06 17:05:43 INFO Master: Launching executor app-20170105220118-0005/34948 on worker worker-20170105215335-10.251.228.103-28641
17/01/06 17:05:44 INFO Master: Removing executor app-20170105220118-0005/34947 because it is EXITED
17/01/06 17:05:44 INFO Master: Launching executor app-20170105220118-0005/34949 on worker worker-20170105215345-10.251.228.107-53462
17/01/06 17:05:46 INFO Master: Removing executor app-20170105220118-0005/34946 because it is EXITED
17/01/06 17:05:46 INFO Master: Launching executor app-20170105220118-0005/34950 on worker worker-20170105215332-10.251.228.102-23104