Question

我正在运行Spark 2.0.2和Mesos 0.28.2。

我尝试使用ZooKeeper管理的Mesos群集作为主程序向Spark提交应用程序：

$SPARK_HOME/bin/spark-submit --verbose \
--conf spark.mesos.executor.docker.image=$DOCKER_IMAGE \
--conf spark.mesos.executor.home=$SPARK_HOME \
--conf spark.executorEnv.MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos.so \
--deploy-mode cluster \
--master mesos://zk://<ip 1>:2181,<ip 2>:2181,<ip 3>:2181/mesos \
--class $APP_MAIN_CLASS \
file://$APP_JAR_PATH

（<ip 1>，<ip 2>和<ip 3>是10.0.0.0/8块中的IPv4地址）

根据documentation，我似乎有适合主人的格式：

Mesos的主URL的形式为mesos：// host：5050表示单主Mesos群集，或者mesos：// zk：// host1：2181，host2：2181，host3：2181 / mesos for使用ZooKeeper的多主Mesos集群。

但是，看起来Spark正在读取mesos://zk://...字符串，然后尝试连接到zk：

17/04/07 20:10:06 INFO RestSubmissionClient: Submitting a request to launch an application in mesos://zk://<ip 1>:2181,<ip 2>:2181,<ip 3>:2181/mesos.
Exception in thread "main" java.net.UnknownHostException: zk
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at sun.net.NetworkClient.doConnect(NetworkClient.java:180)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:432)
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:527)
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:211)
    at sun.net.www.http.HttpClient.New(HttpClient.java:308)
    at sun.net.www.http.HttpClient.New(HttpClient.java:326)
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)
    at org.apache.spark.deploy.rest.RestSubmissionClient.org$apache$spark$deploy$rest$RestSubmissionClient$$postJson(RestSubmissionClient.scala:214)
    at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$createSubmission$3.apply(RestSubmissionClient.scala:89)
    at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$createSubmission$3.apply(RestSubmissionClient.scala:85)
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
    at org.apache.spark.deploy.rest.RestSubmissionClient.createSubmission(RestSubmissionClient.scala:85)
    at org.apache.spark.deploy.rest.RestSubmissionClient$.run(RestSubmissionClient.scala:417)
    at org.apache.spark.deploy.rest.RestSubmissionClient$.main(RestSubmissionClient.scala:430)
    at org.apache.spark.deploy.rest.RestSubmissionClient.main(RestSubmissionClient.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

如何让Spark认识到它应该使用三个ZooKeeper节点而不是尝试连接到不存在的zk主机？

Answer 1

tl; dr 除非您将--deploy-mode更改为client或使用包含单个Mesos主机的主网址，否则它将无效。 mesos://host:port。

以下行给出提示在哪里可以找到相关代码。

17/04/07 20:10:06 INFO RestSubmissionClient：在mesos：// zk：//：2181，：2181，：2181 / mesos中提交启动应用程序的请求。

看起来该消息仅针对使用Spark Standalone和Apache Mesos的--deploy-mode cluster打印出来。将其更改为默认client，部署路径将更改，并希望接受主URL。

自己查看负责群集部署的代码 - RestSubmissionClient。

Here RestSubmissionClient说：

private val supportedMasterPrefixes = Seq("spark://", "mesos://")

证明了mesos://个网址，但here您会看到以下内容：

private val masters: Array[String] = if (master.startsWith("spark://")) {
  Utils.parseStandaloneMasterUrls(master)
} else {
  Array(master)
}

打印出here，因为上面的INFO消息显示的URL只能是一个Mesos主页。

如何激活提交给ZooKeeper管理的Mesos集群（给出java.net.UnknownHostException：zk for mesos：// zk：// master URL）？

1 个答案: