Spark连接到独立集群上的slave

时间:2021-01-12 13:22:14

标签: apache-spark

所以我在我的 Mac 上有我的主节点,我可以在网络服务器上检查我的主 url 为 spark://private_ip_address:7077。然后我尝试连接远程服务器上的从节点。

所以我从主服务器运行 start-slaves.sh 并在远程服务器上创建一个日志。所以我假设 ssh 连接没问题。但是如日志内容所示,无法连接到master。

21/01/12 13:06:36 INFO ResourceUtils: ==============================================================
21/01/12 13:06:36 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
21/01/12 13:06:36 INFO WorkerWebUI: Bound WorkerWebUI to 127.0.0.0, and started at http://127.0.0.0:8081
21/01/12 13:06:36 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:06:48 INFO Worker: Retrying connection to master (attempt # 1)
21/01/12 13:06:48 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:00 INFO Worker: Retrying connection to master (attempt # 2)
21/01/12 13:07:00 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:12 INFO Worker: Retrying connection to master (attempt # 3)
21/01/12 13:07:12 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:24 INFO Worker: Retrying connection to master (attempt # 4)
21/01/12 13:07:24 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:36 INFO Worker: Retrying connection to master (attempt # 5)
21/01/12 13:07:36 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:07:48 INFO Worker: Retrying connection to master (attempt # 6)
21/01/12 13:07:48 INFO Worker: Connecting to master master_node_private_ip_address:7077...
21/01/12 13:08:36 ERROR RpcOutboxMessage: Ask terminated before connecting successfully
21/01/12 13:08:36 WARN NettyRpcEnv: Ignored failure: java.io.IOException: Connecting to /master_node_private_ip_address:7077 timed out (120000 ms)
21/01/12 13:08:36 WARN Worker: Failed to connect to master master_node_private_ip_address:7077
org.apache.spark.SparkException: Exception thrown in awaitResult: 
    at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:302)
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:75)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRefByURI(RpcEnv.scala:101)
    at org.apache.spark.rpc.RpcEnv.setupEndpointRef(RpcEnv.scala:109)
    at org.apache.spark.deploy.worker.Worker$$anon$1.run(Worker.scala:277)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Connecting to /master_node_private_ip_address:7077 timed out (120000 ms)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:251)
    at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:195)
    at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:204)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:202)
    at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:198)
    ... 4 more

如果您能帮助我理解为什么无法正确连接并在网络服务器上显示从属节点,我将不胜感激。

0 个答案:

没有答案