对于我来说,Spark隐藏的REST API(https://gist.github.com/arturmkrtchyan/5d8559b2911ac951d34a)对于提交Scala作业非常有用。但是有什么方法可以通过此API提交SparkR作业吗?
我尝试过,但是遇到了这个错误:
Exception in thread "main" java.lang.ClassNotFoundException:
它正在寻找一个主班,但是自然地,R不会有主班。我只是像传递spark-submit
一样传递R文件。以下是请求有效负载:
{
"action" : "CreateSubmissionRequest",
"appArgs" : [],
"appResource" : "file:/home/piyush/global_ui_pipeline.r",
"clientSparkVersion" : "2.3.1",
"environmentVariables" : {
"SPARK_ENV_LOADED" : "1"
},
"mainClass" : "",
"sparkProperties" : {
"spark.driver.supervise" : "false",
"spark.app.name" : "sparkR",
"spark.eventLog.enabled": "true",
"spark.submit.deployMode" : "cluster",
"spark.master" : "spark://localhost:6066"
}
}
如果我从json中删除mainClass字段,则会出现以下错误:
Malformed request: org.apache.spark.deploy.rest.SubmitRestMissingFieldException: Main class is missing.
隐藏的rest API是否可以用于提交sparkR作业?如果可以的话,怎么办?