Spark& amp;纱。我正在Yarn(cloudera)上运行一个火花应用程序。一切正常,但如果我尝试缓存RDD,那么作业会在处理过程中冻结。
RDD占用3GB。我正在试图弄清楚如何解决这个问题。显然必须存在资源争用......
16/05/17 19:49:41 INFO ExecutorAllocationManager: Requesting 16 new executors because tasks are backlogged (new desired total will be 72)
16/05/17 19:49:42 INFO ExecutorAllocationManager: Requesting 32 new executors because tasks are backlogged (new desired total will be 104)
16/05/17 19:49:43 INFO ExecutorAllocationManager: Requesting 64 new executors because tasks are backlogged (new desired total will be 168)
16/05/17 19:49:44 INFO ExecutorAllocationManager: Requesting 128 new executors because tasks are backlogged (new desired total will be 296)
16/05/17 19:49:45 INFO ExecutorAllocationManager: Requesting 172 new executors because tasks are backlogged (new desired total will be 468)
......它只是挂在那里......