ClosedChannelException Spark中的错误

时间:2017-02-01 17:03:23

标签: scala apache-spark amazon-emr

我有用scala编写的spark代码。我们在Amazon EMR中执行代码。

我在跑步时遇到错误。

17/02/01 04:06:23 ERROR TransportClient: Failed to send RPC 7076672670423246515 to ip-XXX-XX-XX-XXX.ec2.internal/XXX.XX.XX.XXX:33340: java.nio.channels.ClosedChannelException
java.nio.channels.ClosedChannelException
17/02/01 04:06:23 WARN NettyRpcEndpointRef: Error sending message [message = KillExecutors(List(3))] in 3 attempts
java.io.IOException: Failed to send RPC 7076672670423246515 to ip-XXX-XX-XX-XXX.ec2.internal/XXX.XX.XX.XXX:33340: java.nio.channels.ClosedChannelException
    at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:239)
    at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:226)
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
    at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:567)
    at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:424)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:745)
    at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:646)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1054)
    at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:658)
    at io.netty.channel.AbstractChannelHandlerContext.access$2000(AbstractChannelHandlerContext.java:32)
    at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.write(AbstractChannelHandlerContext.java:939)
    at io.netty.channel.AbstractChannelHandlerContext$WriteAndFlushTask.write(AbstractChannelHandlerContext.java:991)
    at io.netty.channel.AbstractChannelHandlerContext$AbstractWriteTask.run(AbstractChannelHandlerContext.java:924)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:380)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
17/02/01 04:06:23 ERROR YarnSchedulerBackend$YarnSchedulerEndpoint: Sending KillExecutors(List(3)) to AM was unsuccessful
org.apache.spark.SparkException: Error sending message [message = KillExecutors(List(3))]
    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:118)
    at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:77)
    at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$4.apply$mcV$sp(YarnSchedulerBackend.scala:200)
    at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$4.apply(YarnSchedulerBackend.scala:200)
    at org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint$$anonfun$receiveAndReply$1$$anonfun$applyOrElse$4.apply(YarnSchedulerBackend.scala:200)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
    at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

我检查过没有。在JAVA 7以及JAVA 8中论坛和尝试编译代码。它仍然抛出相同的错误。

0 个答案:

没有答案