错误:java.io.IOException:无法发送RPC java.nio.channels.ClosedChannelException

时间:2017-02-16 05:16:05

标签: amazon-web-services apache-spark yarn emr

我在AWS EMR上运行Spark程序。当spark尝试执行SparkSQL查询时,我得到了这个问题末尾提到的错误。自联接导致交叉连接,如结果,这是一个巨大的结果。这是内存问题吗?如果是这样,是否需要升级我的集群才能运行它?由于我没有时间,我会很感激快速回答。提前致谢。 错误讯息:

java.io.IOException: Failed to send RPC 7911398405091204769 to /10.31.240.189:34032: java.nio.channels.ClosedChannelException
at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:801)
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:699)
at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1122)
at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:633)
at io.netty.channel.AbstractChannelHandlerContext.access$1900(AbstractChannelHandlerContext.java:32)
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:908)
at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:960)
at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:893)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
17/02/16 04:23:34 ERROR YarnScheduler: Lost executor 26 on ip-10-31-240-189.ec2.internal: Slave lost
17/02/16 04:23:34 WARN TaskSetManager: Lost task 72.0 in stage 12.0 (TID 1282, ip-10-31-240-189.ec2.internal): ExecutorLostFailure (executor 26 exited caused by one of the running tasks) Reason: Slave lost 

0 个答案:

没有答案