Java作业无法连接到独立本地主服务器

时间：2017-11-28 10:48:36

标签： java apache-spark

我是新手，并且一直试图通过一个独立的本地主人来运行我的第一个java spark工作。 现在我的主人起来了，一个工作人员也被注册了，但是当在火花程序下运行时，我得到了org.apache.spark.SparkException：在awaitResult中抛出了异常。 当master设置为local时，我的程序应该正常工作。

My Spark Code：

public static void main(String[] args) {

    //Setup configuration
    String appName = "My Very First Spark Job";
    //String sparkMaster = "local[2]";
    String sparkMaster = "spark://10.0.0.116:7077";

    JavaSparkContext spContext = null;

    SparkConf conf = new SparkConf()
            .setAppName(appName)
            .setMaster(sparkMaster);

    //Create Spark Context from configuration
    spContext = new JavaSparkContext(conf);

日志：

    17/11/28 21:22:23 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://10.0.0.116:7077...
    17/11/28 21:22:23 INFO TransportClientFactory: Successfully created connection to /10.0.0.116:7077 after 30 ms (0 ms spent in bootstraps)
    17/11/28 21:22:23 WARN StandaloneAppClient$ClientEndpoint: Failed to connect to master 10.0.0.116:7077
    org.apache.spark.SparkException: Exception thrown in awaitResult
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59)
        at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167)
        at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:88)
        at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:96)
        at org.apache.spark.deploy.client.StandaloneAppClient$ClientEndpoint$$anonfun$tryRegisterAllMasters$1$$anon$1.run(StandaloneAppClient.scala:109)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
    Caused by: java.lang.RuntimeException: java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:197)
        at java.io.DataInputStream.readUTF(DataInputStream.java:609)
        at java.io.DataInputStream.readUTF(DataInputStream.java:564)
        at org.apache.spark.rpc.netty.RequestMessage$.readRpcAddress(NettyRpcEnv.scala:582)
        at org.apache.spark.rpc.netty.RequestMessage$.apply(NettyRpcEnv.scala:592)
        at org.apache.spark.rpc.netty.NettyRpcHandler.internalReceive(NettyRpcEnv.scala:651)
        at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:636)
        at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157)
        at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)
        at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
        at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:748)

        at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:190)
        at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:121)
        at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
        at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
        at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
        ... 1 more

Spark Master：

    Jings-MBP-6:bin jingzhou$ ./spark-class org.apache.spark.deploy.master.Master
    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
    17/11/28 20:55:11 INFO Master: Started daemon with process name: 12707@Jings-MBP-6.gateway
    17/11/28 20:55:11 INFO SignalUtils: Registered signal handler for TERM
    17/11/28 20:55:11 INFO SignalUtils: Registered signal handler for HUP
    17/11/28 20:55:11 INFO SignalUtils: Registered signal handler for INT
    17/11/28 20:55:11 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    17/11/28 20:55:11 INFO SecurityManager: Changing view acls to: jingzhou
    17/11/28 20:55:11 INFO SecurityManager: Changing modify acls to: jingzhou
    17/11/28 20:55:11 INFO SecurityManager: Changing view acls groups to: 
    17/11/28 20:55:11 INFO SecurityManager: Changing modify acls groups to: 
    17/11/28 20:55:11 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jingzhou); groups with view permissions: Set(); users  with modify permissions: Set(jingzhou); groups with modify permissions: Set()
    17/11/28 20:55:12 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
    17/11/28 20:55:12 INFO Master: Starting Spark master at spark://10.0.0.116:7077
    17/11/28 20:55:12 INFO Master: Running Spark version 2.2.0
    17/11/28 20:55:12 INFO Utils: Successfully started service 'MasterUI' on port 8080.
    17/11/28 20:55:12 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://10.0.0.116:8080
    17/11/28 20:55:12 INFO Utils: Successfully started service on port 6066.
    17/11/28 20:55:12 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066
    17/11/28 20:55:12 INFO Master: I have been elected leader! New state: ALIVE
    17/11/28 20:59:27 INFO Master: Registering worker 10.0.0.116:64461 with 8 cores, 15.0 GB RAM
    17/11/28 21:03:42 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() on RPC id 4722074090999773956
    java.io.EOFException
        at java.io.DataInputStream.readFully(DataInputStream.java:197)
        at java.io.DataInputStream.readUTF(DataInputStream.java:609)
        at java.io.DataInputStream.readUTF(DataInputStream.java:564)
        at org.apache.spark.rpc.netty.RequestMessage$.readRpcAddress(NettyRpcEnv.scala:582)
        at org.apache.spark.rpc.netty.RequestMessage$.apply(NettyRpcEnv.scala:592)
        at org.apache.spark.rpc.netty.NettyRpcHandler.internalReceive(NettyRpcEnv.scala:651)
        at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:636)
        at org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:157)
        at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:105)
        at org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
        at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:748)

Spark Worker：

    Jings-MBP-6:bin jingzhou$ ./spark-class org.apache.spark.deploy.worker.Worker spark://10.0.0.116:7077
    Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
    17/11/28 20:59:26 INFO Worker: Started daemon with process name: 12794@Jings-MBP-6.gateway
    17/11/28 20:59:26 INFO SignalUtils: Registered signal handler for TERM
    17/11/28 20:59:26 INFO SignalUtils: Registered signal handler for HUP
    17/11/28 20:59:26 INFO SignalUtils: Registered signal handler for INT
    17/11/28 20:59:26 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    17/11/28 20:59:26 INFO SecurityManager: Changing view acls to: jingzhou
    17/11/28 20:59:26 INFO SecurityManager: Changing modify acls to: jingzhou
    17/11/28 20:59:26 INFO SecurityManager: Changing view acls groups to: 
    17/11/28 20:59:26 INFO SecurityManager: Changing modify acls groups to: 
    17/11/28 20:59:26 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jingzhou); groups with view permissions: Set(); users  with modify permissions: Set(jingzhou); groups with modify permissions: Set()
    17/11/28 20:59:27 INFO Utils: Successfully started service 'sparkWorker' on port 64461.
    17/11/28 20:59:27 INFO Worker: Starting Spark worker 10.0.0.116:64461 with 8 cores, 15.0 GB RAM
    17/11/28 20:59:27 INFO Worker: Running Spark version 2.2.0
    17/11/28 20:59:27 INFO Worker: Spark home: /Users/jingzhou/Desktop/hadoop/spark/spark-2.2.0-bin-hadoop2.7
    17/11/28 20:59:27 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
    17/11/28 20:59:27 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://10.0.0.116:8081
    17/11/28 20:59:27 INFO Worker: Connecting to master 10.0.0.116:7077...
    17/11/28 20:59:27 INFO TransportClientFactory: Successfully created connection to /10.0.0.116:7077 after 26 ms (0 ms spent in bootstraps)
    17/11/28 20:59:27 INFO Worker: Successfully registered with master spark://10.0.0.116:7077

1 个答案:

答案 0 :(得分：2)

当我尝试从我的eclipse开发环境连接到本地集群时遇到了同样的问题。

在我的情况下，它与master和我的开发环境版本不匹配，我在使用maven依赖项。

群集是：2.2.1 我的开发人员是：2.1.0

我更正版本后，错误已解决。

相关问题
最新问题