Question

我已通过以下链接设置了火花集群：

https://github.com/big-data-europe/docker-spark

Docker-compose看起来像这样：

version: '2'
services:
  spark-master:
    image: bde2020/spark-master:2.4.4-hadoop2.7
    container_name: spark-master
    ports:
      - "8080:8080"
      - "7077:7077"
      - "6066:6066"
    environment:
      - INIT_DAEMON_STEP=setup_spark
      - SPARK_CONF_DIR=/spark/conf
    volumes:
      - ./spark/conf:/spark/conf
  spark-worker-1:
    image: bde2020/spark-worker:2.4.4-hadoop2.7
    container_name: spark-worker-1
    depends_on:
      - spark-master
    ports:
      - "8081:8081"
    environment:
      - "SPARK_MASTER=spark://spark-master:7077"

我已启用rest api作为火花。

我提交了火花作业：

curl -X POST http://localhost:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
  "action" : "CreateSubmissionRequest",
  "appArgs" : ["file://C:/Input.json"],
  "appResource" : "file://C:/test.jar",
  "clientSparkVersion" : "2.4.4",
  "environmentVariables" : {
    "SPARK_ENV_LOADED" : "1"
  },
  "mainClass" : "SampleApplication",
  "sparkProperties" : {
    "spark.driver.supervise" : "false",
    "spark.app.name" : "TestJob",
    "spark.eventLog.enabled": "true",
    "spark.submit.deployMode" : "cluster",
    "spark.master" : "spark://localhost:6066"
  }
}'

这是响应：

{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20191205021458-0002",
  "serverSparkVersion" : "2.4.4",
  "submissionId" : "driver-20191205021458-0002",
  "success" : true
}

当我使用以下方法检查提交状态时：

curl http://localhost:6066/v1/submissions/status/driver-20191205021458-0002

我看到驱动程序状态为ERROR。我无法调试为什么会出现该错误

{
  "action" : "SubmissionStatusResponse",
  "driverState" : "ERROR",
  "message" : "Exception from the cluster:\njava.lang.IllegalArgumentException: URI has an authority component\n\tjava.io.File.<init>(File.java:423)\n\torg.apache.spark.util.Utils$.doFetchFile(Utils.scala:718)\n\torg.apache.spark.util.Utils$.fetchFile(Utils.scala:509)\n\torg.apache.spark.deploy.worker.DriverRunner.downloadUserJar(DriverRunner.scala:155)\n\torg.apache.spark.deploy.worker.DriverRunner.prepareAndRunDriver(DriverRunner.scala:173)\n\torg.apache.spark.deploy.worker.DriverRunner$$anon$1.run(DriverRunner.scala:92)",
  "serverSparkVersion" : "2.4.4",
  "submissionId" : "driver-20191205021458-0002",
  "success" : true,
  "workerHostPort" : "172.21.0.3:39859",
  "workerId" : "worker-20191205015200-172.21.0.3-39859"
}

Spark作业的提交状态响应错误

0 个答案: