Question

我正在尝试将配置信息传递给Amazon EMR中的Spark，如下所示

spark-submit --jars "/home/hadoop/transfer_cluster/run_spark/spark_jars/jars/trove-3.0.2.jar" --class SparkPTE bin/pte_sc.jar arabic_undirected -–conf spark.yarn.nodemanager.vmem-check-enabled=false

但我收到以下错误，因为spark无法解析我的配置信息。

18/04/06 07:48:22 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
Exception in thread "main" java.lang.NumberFormatException: For input string: "-–conf"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:580)
    at java.lang.Integer.parseInt(Integer.java:615)
    at SparkPTE.sparkContext(SparkPTE.java:91)
    at SparkPTE.main(SparkPTE.java:79)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:775)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

如果我在--jar之前给出--config，我会收到以下错误。

spark-submit -–conf spark.yarn.nodemanager.vmem-check-enabled=false --jars "/home/hadoop/transfer_cluster/run_spark/spark_jars/jars/trove-3.0.2.jar" --class SparkPTE bin/pte_sc.jar arabic_undirected


Error: Unrecognized option: -–conf

以下为我工作

spark-submit --conf spark.yarn.nodemanager.vmem-check-enabled=false --jars "/home/hadoop/transfer_cluster/run_spark/spark_jars/jars/trove-3.0.2.jar" --class SparkPTE bin/pte_sc.jar arabic_undirected

Answer 1

您需要在您尝试运行的jar名称之前提供--conf选项。这是因为你在jar名称之后写的任何内容都将作为该jar的参数。

传递--config时Spark-submit失败

1 个答案: