Question

我有一个很小的spark scala工作，需要几个参数，即 http_endpoint

import scalaj.http.{Http, HttpOptions}

1 object Main extends App {
2
3   val master = args(0)
4   val http_endpoint = args(1)
5   ...
6   ...
7   
8   def sendRequest = ( text: String ) => {
9      println("#####################: " + http_endpoint)
10     val result = Http(http_endpoint)
11    ...
12    ...
13  }

然后我将其作为火花提交作业发送给纱线。

spark-submit \
--master yarn \
my-job-example-assembly-1.0.0-SNAPSHOT.jar \
yarn \
"http://my-url-example-12311.us-west-1.elb.amazonaws.com/my/app"

错误：

#####################: null
Caused by: java.net.MalformedURLException: no protocol: null
    at java.net.URL.<init>(URL.java:611)
    at java.net.URL.<init>(URL.java:508)
    at java.net.URL.<init>(URL.java:457)
    at scalaj.http.HttpRequest.exec(Http.scala:343)
    at scalaj.http.HttpRequest.asString(Http.scala:492)

我什至在发送的URL周围使用了语音引号，但失败了。当我在第10行的代码中对此进行硬编码时，它可以工作。知道作为arg传递时我如何仍能使它正常工作吗？

Answer 1

尝试使用--conf spark.driver.args="<args>"

传递参数

在您的代码中，使用spark.sparkContext.getConf.get()获取参数。

spark-submit --master yarn --conf spark.driver.args="yarn http://my-url-example-12311.us-west-1.elb.amazonaws.com/my/app" my-job-example-assembly-1.0.0-SNAPSHOT.jar

Example:

spark-shell --conf spark.driver.args="yarn http://my-url-example-12311.us-west-1.elb.amazonaws.com/my/app"

val arg0=spark.sparkContext.getConf.get("spark.driver.args").split("\\s+")(0)
//String = yarn
val arg1=spark.sparkContext.getConf.get("spark.driver.args").split("\\s+")(1)
//String = http://my-url-example-12311.us-west-1.elb.amazonaws.com/my/app

java.net.MalformedURLException：无协议：将arg传递给spark-submit作业时为null

1 个答案: