超出Spark GC开销限制

Question

我正在尝试使用hivecontext加载一个巨大的镶木桌子的分区。分区的大小为500 GB。但每次出现以下错误都会失败。我在cloudera环境中使用spark 1.3。

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Exception in thread "Yarn application state monitor" java.lang.OutOfMemoryError: GC overhead limit exceeded
16/02/12 11:44:37 ERROR ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-akka.actor.default-dispatcher-20] shutting down ActorSystem [sparkDriver]
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at akka.actor.FSM$class.makeTransition(FSM.scala:617)
        at akka.remote.EndpointWriter.makeTransition(Endpoint.scala:443)
        at akka.actor.FSM$class.applyState(FSM.scala:605)
        at akka.remote.EndpointWriter.applyState(Endpoint.scala:443)
        at akka.actor.FSM$class.processEvent(FSM.scala:600)
        at akka.remote.EndpointWriter.processEvent(Endpoint.scala:443)
        at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:589)
        at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:561)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
        at akka.actor.ActorCell.invoke(ActorCell.scala:456)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
16/02/12 11:44:40 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/02/12 11:44:42 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/02/12 11:44:56 ERROR Remoting: Remoting error: [Failure during shutdown of remoting.] [
akka.remote.RemoteTransportException: Failure during shutdown of remoting.
        at akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:129)
        at akka.remote.Remoting$$anonfun$shutdown$1.applyOrElse(Remoting.scala:150)
        at akka.remote.Remoting$$anonfun$shutdown$1.applyOrElse(Remoting.scala:142)
        at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:434)
        at scala.concurrent.Future$$anonfun$andThen$1.apply(Future.scala:433)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at scala.concurrent.impl.ExecutionContextImpl$$anon$3.exec(ExecutionContextImpl.scala:107)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: akka.pattern.AskTimeoutException: Timed out
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:334)
        at akka.actor.Scheduler$$anon$11.run(Scheduler.scala:118)
        at scala.concurrent.Future$InternalCallbackExecutor$.scala$concurrent$Future$InternalCallbackExecutor$$unbatchedExecute(Future.scala:694)
        at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:691)
        at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:455)
        at akka.actor.LightArrayRevolverScheduler$$anon$12.executeBucket$1(Scheduler.scala:407)
        at akka.actor.LightArrayRevolverScheduler$$anon$12.nextTick(Scheduler.scala:411)
        at akka.actor.LightArrayRevolverScheduler$$anon$12.run(Scheduler.scala:363)
        at java.lang.Thread.run(Thread.java:745)