我一直使用我的Spark程序获得CancelledKeyExpection
和ClosedChannelException
。我尝试通过将spark.network.timeout
增加到600并将spark.akka.frameSize
增加到1280来补救它。此外,我已经为每个执行程序和驱动程序分配了足够的内存。另外,我使用Spark的最新版本,即Spark 1.5。我甚至试过1.4,但得到了同样的错误。那么,可能是什么问题呢?它让我疯了!错误如下所示。
java.nio.channels.CancelledKeyException
at org.apache.spark.network.nio.ConnectionManager.run(ConnectionManager.scala:461)
at org.apache.spark.network.nio.ConnectionManager$$anon$7.run(ConnectionManager.scala:193)
15/09/16 16:27:38 INFO nio.ConnectionManager: Removing ReceivingConnection to ConnectionManagerId(bigdata-03.cluster,50585)
15/09/16 16:27:38 INFO nio.ConnectionManager: Removing SendingConnection to ConnectionManagerId(bigdata-03.cluster,50585)
15/09/16 16:27:38 ERROR nio.SendingConnection: Exception while reading SendingConnection to ConnectionManagerId(bigdata-03.cluster,50585)
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
at org.apache.spark.network.nio.SendingConnection.read(Connection.scala:421)
at org.apache.spark.network.nio.ConnectionManager$$anon$9.run(ConnectionManager.scala:259)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
15/09/16 16:27:38 ERROR nio.ConnectionManager: Exception while sending message.
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:257)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:300)
at org.apache.spark.network.nio.SendingConnection.read(Connection.scala:421)
at org.apache.spark.network.nio.ConnectionManager$$anon$9.run(ConnectionManager.scala:259)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)