为什么在申请新执行人时会引发应用程序冻结?

时间:2016-05-17 19:51:18

标签: apache-spark yarn cloudera cloudera-cdh

Spark& amp;纱。我正在Yarn(cloudera)上运行一个火花应用程序。一切正常,但如果我尝试缓存RDD,那么作业会在处理过程中冻结。

RDD占用3GB。我正在试图弄清楚如何解决这个问题。显然必须存在资源争用......

16/05/17 19:49:41 INFO ExecutorAllocationManager: Requesting 16 new executors because tasks are backlogged (new desired total will be 72)
16/05/17 19:49:42 INFO ExecutorAllocationManager: Requesting 32 new executors because tasks are backlogged (new desired total will be 104)
16/05/17 19:49:43 INFO ExecutorAllocationManager: Requesting 64 new executors because tasks are backlogged (new desired total will be 168)
16/05/17 19:49:44 INFO ExecutorAllocationManager: Requesting 128 new executors because tasks are backlogged (new desired total will be 296)
16/05/17 19:49:45 INFO ExecutorAllocationManager: Requesting 172 new executors because tasks are backlogged (new desired total will be 468)

......它只是挂在那里......

0 个答案:

没有答案