Question

我正在使用spark 0.90独立模式。

当我在独立模式下尝试使用流媒体应用程序时，我收到连接拒绝例外。

我在/etc/hosts中添加了主机名，也尝试使用IP。在这两种情况下，工人都在没有任何问题的情况下注册了主人。

有没有办法解决这个问题？

14/02/28 07:15:01 INFO Master: akka.tcp://driverClient@127.0.0.1:55891 got disassociated, removing it.
14/02/28 07:15:04 INFO Master: Registering app Twitter Streaming
14/02/28 07:15:04 INFO Master: Registered app Twitter Streaming with ID app-20140228071504-0000
14/02/28 07:34:42 INFO Master: akka.tcp://spark@127.0.0.1:33688 got disassociated, removing it.
14/02/28 07:34:42 INFO LocalActorRef: Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://sparkMaster/deadLetters] to Actor[akka://sparkMaster/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FsparkMaster%4010.165.35.96%3A38903-6#-1146558090] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
14/02/28 07:34:42 ERROR EndpointWriter: AssociationError [akka.tcp://sparkMaster@10.165.35.96:8910] -> [akka.tcp://spark@127.0.0.1:33688]: Error [Association failed with [akka.tcp://spark@127.0.0.1:33688]] [
akka.remote.EndpointAssociationException: Association failed with [akka.tcp://spark@127.0.0.1:33688]
Caused by: akka.remote.transport.netty.NettyTransport$$anonfun$associate$1$$anon$2: Connection refused: /127.0.0.1:33688

Answer 1

在群集模式下运行Spark时遇到类似问题。我的问题是服务器是以主机名'fluentd：7077'而不是FQDN启动的。我编辑了

/sbin/start-master.sh

反映我的远程节点如何与-ip标志连接。

/usr/lib/jvm/jdk1.7.0_51/bin/java -cp :/home/vagrant/spark-0.9.0-incubating-bin-   hadoop2/conf:/home/vagrant/spark-0.9.0-incuba
ting-bin-hadoop2/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar -Dspark.akka.logLifecycleEvents=true -Djava.library.path= -Xms512m -Xmx512m org.ap
ache.spark.deploy.master.Master --ip fluentd.alex.dev --port 7077 --webui-port 8080

希望这有帮助。

Spark 0.90独立连接被拒绝

1 个答案: