Question

请原谅这个愚蠢的问题。 Jupyter笔记本中的代码是否可以提交给Mesos集群？

这很好：

./bin/spark-submit --class org.apache.spark.examples.SparkPi --master mesos://master.tld:7077 --deploy-mode cluster --executor-memory 12G --total-executor-cores 10 --conf 'spark.mesos.executor.home=/opt/spark' http://172.18.1.29:8000/spark-examples-1.0.jar

，并从我提交作业的位置得到以下HTTP请求和响应：

Content-Type: application/json
charset: utf-8
User-Agent: Java/1.8.0_171
Host: master.tld:7077
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 757

{
  "action" : "CreateSubmissionRequest",
  "appArgs" : [ "100" ],
  "appResource" : "http://172.18.1.29:8000/spark-examples-1.0.jar",
  "clientSparkVersion" : "2.3.2",
  "environmentVariables" : {
    "SPARK_SCALA_VERSION" : "2.12",
    "SPARK_CONF_DIR" : "/opt/spark-2.3.2-bin-hadoop2.7/conf"
  },
  "mainClass" : "org.apache.spark.examples.SparkPi",
  "sparkProperties" : {
    "spark.jars" : "http://172.18.1.29:8000/spark-examples-1.0.jar",
    "spark.driver.supervise" : "false",
    "spark.app.name" : "org.apache.spark.examples.SparkPi",
    "spark.cores.max" : "10",
    "spark.submit.deployMode" : "cluster",
    "spark.master" : "mesos://master.tld:7077",
    "spark.executor.memory" : "12G",
    "spark.mesos.executor.home" : "/opt/spark"
  }
}HTTP/1.1 200 OK
Date: Tue, 11 Dec 2018 14:56:08 GMT
Content-Type: application/json;charset=utf-8
Content-Length: 146
Server: Jetty(9.3.z-SNAPSHOT)

{
  "action" : "CreateSubmissionResponse",
  "serverSparkVersion" : "2.3.2",
  "submissionId" : "driver-20181211155608-0003",
  "success" : true
}

但是，我启动像这样的docker run -p 8888:8888 --network host -e SPARK_OPTS='--master=mesos://master.tld:7077' jupyter/all-spark-notebook的jupyter笔记本，以下代码永远不会返回：

val count = sc.parallelize(1 to 100).filter { _ =>
  val x = math.random
  val y = math.random
  x*x + y*y < 1
}.count()

println(s"Pi is roughly ${4.0 * count / 100}")

该单元永远不会返回并导致以下HTTP请求和响应：

User-Agent: libprocess/scheduler-c6cf51ad-409d-410b-b460-185731b51162@172.17.0.2:45269
Libprocess-From: scheduler-c6cf51ad-409d-410b-b460-185731b51162@172.17.0.2:45269
Connection: Keep-Alive
Host: 
Transfer-Encoding: chunked

...I
G
.jovyan..Apache Toree!........:.79fb047728edJ.http://79fb047728ed:4040HTTP/1.1 468 468
Date: Tue, 11 Dec 2018 12:09:38 GMT
Content-Type: application/json;charset=utf-8
Content-Length: 224
Server: Jetty(9.3.z-SNAPSHOT)

{
  "action" : "ErrorResponse",
  "highestProtocolVersion" : "v1",
  "message" : "Unknown protocol version 'master'. Please submit requests through http://[host]:[port]/v1/submissions/...",
  "serverSparkVersion" : "2.3.2"
}```

将Jupyter Notebook代码提交到Mesos集群

0 个答案: