Question

我使用PC作为Spark服务器，同时又与Spark Worker一起使用Spark 2.3.1。

首先，我使用了Ubuntu 16.04 LTS。一切正常，我尝试运行SparkPi示例（使用spark-submit和spark-shell），并且能够正常运行。我也尝试使用Spark的REST API和以下POST字符串来运行它：

curl -X POST http://192.168.1.107:6066/v1/submissions/create --header "Content-Type:application/json" --data '{
  "action": "CreateSubmissionRequest",
  "appResource": "file:/home/Workspace/Spark/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar",
  "clientSparkVersion": "2.3.1",
  "appArgs": [ "10" ],
  "environmentVariables" : {
    "SPARK_ENV_LOADED" : "1"
  },
  "mainClass": "org.apache.spark.examples.SparkPi",
  "sparkProperties": {
    "spark.jars": "file:/home/Workspace/Spark/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar",
    "spark.driver.supervise":"false",
    "spark.executor.memory": "512m",
    "spark.driver.memory": "512m",
    "spark.submit.deployMode":"cluster",
    "spark.app.name": "SparkPi",
    "spark.master": "spark://192.168.1.107:7077"
  }
}'

在对此进行了测试之后，我必须转到Windows，因为无论如何都将在Windows上完成。我能够（手动）运行服务器和辅助服务器，添加winutils.exe，还使用spark-shell和spark-submit运行SparkPi示例，所有功能也都可以运行。问题是当我通过以下POST字符串使用REST API时：

curl -X POST http://192.168.1.107:6066/v1/submissions/create --header "Content-Type:application/json" --data '{
      "action": "CreateSubmissionRequest",
      "appResource": "file:D:/Workspace/Spark/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar",
      "clientSparkVersion": "2.3.1",
      "appArgs": [ "10" ],
      "environmentVariables" : {
        "SPARK_ENV_LOADED" : "1"
      },
      "mainClass": "org.apache.spark.examples.SparkPi",
      "sparkProperties": {
        "spark.jars": "file:D:/Workspace/Spark/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar",
        "spark.driver.supervise":"false",
        "spark.executor.memory": "512m",
        "spark.driver.memory": "512m",
        "spark.submit.deployMode":"cluster",
        "spark.app.name": "SparkPi",
        "spark.master": "spark://192.168.1.107:7077"
      }
    }'

只有路径有些不同，但是我的工作人员总是失败。日志说：

"Exception from the cluster: java.lang.NullPointerException                                                
org.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:151)
org.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scal173)
org.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:92)"

我搜索了，但还没有解决方案。

Answer 1

所以，终于我找到了原因。

我从以下来源阅读源代码： https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/worker/DriverRunner.scala

通过检查，我认为问题不是出自Spark，但参数未正确读取。这意味着我以错误的方式设置了参数格式。

因此，在尝试了几件事之后，这是正确的一件事：

appResource": "file:D:/Workspace/Spark/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar"

更改为：

appResource": "file:///D:/Workspace/Spark/spark-2.3.1-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.1.jar"

我对spark.jars参数也做了同样的事情。

这些微小的差异使我花了将近24小时的工作时间... ~~~~

Spark REST API，在Windows上提交应用程序NullPointerException

1 个答案: