尝试连接远程Spark主节点

时间:2016-07-12 05:48:48

标签: scala apache-spark

我已在单独的VM上部署了Spark群集,该VM在单独的物理机器上运行。我已经定义了与主节点相同的计算机。然后我尝试使用远程主节点处理spark作业.Driver应用程序在我的本地计算机上运行。我已经按如下方式定义了Spark conf,

val conf = new SparkConf()
  .setAppName("test app")
  .setMaster("spark://xxx.xxx.xx.xxx:7077")
  .set("spark.deploy.mode", "cluster")
  .set("spark.driver.memory", "1g")

在应用程序运行时,会出现以下错误,

16/07/12 10:01:22 ERROR TransportResponseHandler: Still have 2 requests outstanding when connection from xxx.xxx.xx.xxx/xxx.xxx.xx.xxx7077 is closed
16/07/12 10:01:22 WARN AppClient$ClientEndpoint: Failed to connect to master xxx.xxx.xx.xxx:7077
java.io.IOException: Connection from xxx.xxx.xx.xxx/xxx.xxx.xx.xxx:7077 closed
    at org.apache.spark.network.client.TransportResponseHandler.channelUnregistered(TransportResponseHandler.java:124)
    at org.apache.spark.network.server.TransportChannelHandler.channelUnregistered(TransportChannelHandler.java:94)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
    at io.netty.channel.ChannelInboundHandlerAdapter.channelUnregistered(ChannelInboundHandlerAdapter.java:53)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:158)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelUnregistered(AbstractChannelHandlerContext.java:144)
    at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:739)
    at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:659)
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
    at java.lang.Thread.run(Thread.java:745)

请善意帮我解决这个错误。

0 个答案:

没有答案