我在调试下运行程序，没有发生错误，当我在集群上提交程序时，只运行2分钟，作业自杀了。日志提示如下：
17/05/26 17:52:39 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20170526175239-2059
17/05/26 17:52:39 INFO client.AppClient$ClientEndpoint: Executor added: app-20170526175239-2059/0 on worker-20161221103626-192.168.90.24-34227 (192.168.90.24:34227) with 3 cores
17/05/26 17:52:39 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20170526175239-2059/0 on hostPort 192.168.90.24:34227 with 3 cores, 2.0 GB RAM
17/05/26 17:52:39 INFO client.AppClient$ClientEndpoint: Executor added: app-20170526175239-2059/1 on worker-20161221103626-192.168.90.25-36151 (192.168.90.25:36151) with 3 cores
17/05/26 17:52:39 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20170526175239-2059/1 on hostPort 192.168.90.25:36151 with 3 cores, 2.0 GB RAM
17/05/26 17:52:39 INFO client.AppClient$ClientEndpoint: Executor added: app-20170526175239-2059/2 on worker-20161221103626-192.168.90.23-36295 (192.168.90.23:36295) with 3 cores
17/05/26 17:52:39 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20170526175239-2059/2 on hostPort 192.168.90.23:36295 with 3 cores, 2.0 GB RAM
17/05/26 17:52:39 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44727.
17/05/26 17:52:39 INFO netty.NettyBlockTransferService: Server created on 44727
17/05/26 17:52:39 INFO storage.BlockManagerMaster: Trying to register BlockManager
17/05/26 17:52:39 INFO storage.BlockManagerMasterEndpoint: Registering block manager slave1:44727 with 1247.6 MB RAM, BlockManagerId(driver, slave1, 44727)
17/05/26 17:52:39 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170526175239-2059/1 is now RUNNING
17/05/26 17:52:39 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170526175239-2059/0 is now RUNNING
17/05/26 17:52:39 INFO client.AppClient$ClientEndpoint: Executor updated: app-20170526175239-2059/2 is now RUNNING
17/05/26 17:52:39 INFO storage.BlockManagerMaster: Registered BlockManager
17/05/26 17:52:39 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
17/05/26 17:52:39 INFO executor.Executor: Finished task 0.0 in stage 83.0 (TID 1346). 1391 bytes result sent to driver
17/05/26 17:52:39 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 1350
17/05/26 17:52:39 INFO executor.Executor: Running task 0.0 in stage 101.0 (TID 1350)
17/05/26 17:52:39 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 33
17/05/26 17:52:39 ERROR executor.Executor: Exception in task 0.0 in stage 101.0 (TID 1350)
java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_33_piece0 of broadcast_33
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1222)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:65)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Failed to get broadcast_33_piece0 of broadcast_33
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:137)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:175)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1219)
... 12 more
17/05/26 17:52:39 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 1354
17/05/26 17:52:39 INFO executor.Executor: Running task 0.1 in stage 101.0 (TID 1354)
17/05/26 17:52:39 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 33
17/05/26 17:52:39 ERROR executor.Executor: Exception in task 0.1 in stage 101.0 (TID 1354)
java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_33_piece0 of broadcast_33
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1222)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:65)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Failed to get broadcast_33_piece0 of broadcast_33
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:137)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:175)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1219)
... 12 more
17/05/26 17:52:39 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 1355
17/05/26 17:52:39 INFO executor.Executor: Running task 0.2 in stage 101.0 (TID 1355)
17/05/26 17:52:39 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 33
17/05/26 17:52:39 ERROR executor.Executor: Exception in task 0.2 in stage 101.0 (TID 1355)
java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_33_piece0 of broadcast_33
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1222)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:65)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Failed to get broadcast_33_piece0 of broadcast_33
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:137)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:175)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1219)
... 12 more
17/05/26 17:52:39 INFO executor.CoarseGrainedExecutorBackend: Got assigned task 1356
17/05/26 17:52:39 INFO executor.Executor: Running task 0.3 in stage 101.0 (TID 1356)
17/05/26 17:52:39 INFO broadcast.TorrentBroadcast: Started reading broadcast variable 33
17/05/26 17:52:39 ERROR executor.Executor: Exception in task 0.3 in stage 101.0 (TID 1356)
java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_33_piece0 of broadcast_33
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1222)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:65)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Failed to get broadcast_33_piece0 of broadcast_33
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:137)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:175)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1219)
... 12 more
17/05/26 17:52:39 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown
我可以从HDFS成功加载数据。我的集群是Spark1.6.0和scala2.10.5。 我不知道如何处理这个异常。这个问题的原因是什么？我该怎么处理？