从Rest API提交Sparkr作业

时间:2018-08-30 10:11:29

标签: apache-spark sparkr spark-submit

对于我来说,Spark隐藏的REST API(https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a)对于提交Scala作业非常有用。但是有什么方法可以通过此API提交SparkR作业吗?

我尝试过,但是遇到了这个错误:

Exception in thread "main" java.lang.ClassNotFoundException:

它正在寻找一个主班,但是自然地,R不会有主班。我只是像传递spark-submit一样传递R文件。以下是请求有效负载:

{
  "action" : "CreateSubmissionRequest",
  "appArgs" : [],
  "appResource" : "file:/home/piyush/global_ui_pipeline.r",
  "clientSparkVersion" : "2.3.1",
  "environmentVariables" : {
    "SPARK_ENV_LOADED" : "1"
  },
  "mainClass" : "",
  "sparkProperties" : {
    "spark.driver.supervise" : "false",
    "spark.app.name" : "sparkR",
    "spark.eventLog.enabled": "true",
    "spark.submit.deployMode" : "cluster",
    "spark.master" : "spark://localhost:6066"
  }
}

如果我从json中删除mainClass字段,则会出现以下错误:

Malformed request: org.apache.spark.deploy.rest.SubmitRestMissingFieldException: Main class is missing.

隐藏的rest API是否可以用于提交sparkR作业?如果可以的话,怎么办?

0 个答案:

没有答案