Question

我正在尝试在连接到远程纱线簇的机器中启动spark-job-server。我已按照链接中所述的以下步骤操作：https://github.com/spark-jobserver/spark-jobserver/blob/master/doc/cluster.md

我已经克隆了这个git仓库
更改了spark-jobserver-master \ job-server \ src \ main \ resources \ application.conf的配置
使用sbt构建并以“ job-server / reStart”启动服务器

以下是application.conf的已更改配置

spark {
  master = "yarn"
  submit.deployMode = "cluster"

  jobserver {
    port = 8090
    bind-address = "0.0.0.0"

    context-per-jvm = true
    sqldao{
      jdbc {
        url = "jdbc:h2:tcp://<yarnResournceManagerIP>:9092/h2-db;AUTO_RECONNECT=TRUE"
      }
    }
  }
}
actor {
  remote {
    netty.tcp {
      hostname = "yarnResournceManagerIP" [Ambari home page][1]
      #port = 0
      send-buffer-size = 20 MiB
      receive-buffer-size = 20 MiB
      # This controls the maximum message size, including job results, that can be sent
      maximum-frame-size = 100 MiB
    }
  }
}

启动服务器时出现错误：

[2019-03-07 20:08:52,338] INFO  ka.event.slf4j.Slf4jLogger [] [] - Slf4jLogger started
[2019-03-07 20:08:52,438] INFO  akka.remote.Remoting [] [akka.remote.Remoting] - Starting remoting
[2019-03-07 20:08:52,675] ERROR sport.netty.NettyTransport [] [NettyTransport(akka://JobServer)] - failed to bind to myclusterhostname/10.75.140.166:2552, shutting down Netty transport
[2019-03-07 20:08:52,689] ERROR spark.jobserver.JobServer$ [] [] - Unable to start Spark JobServer: 
org.jboss.netty.channel.ChannelException: Failed to bind to: myclusterhostname/10.75.140.166:2552

Caused by: java.net.BindException: Cannot assign requested address

我的目标是在纱线群集之外运行spark-job-server。我错过了什么吗？

我是否必须提及akka.remote.netty.tcp.port的端口及其应该是什么？
在akka.remote.netty.tcp.hostname中使用我的Yarn群集的资源管理器节点ip错误吗？如果是这样，那应该是什么？

如何配置连接到远程纱线集群的独立Spark-Job-Server

0 个答案: