当我在Amazon AWS EMR中运行我的spark应用程序时，我遇到了以下错误，但我无法理解是什么问题。我创建了一个由两个worker和一个驱动程序节点组成的集群
17/05/30 19:38:34 INFO RMProxy: Connecting to ResourceManager at ip-172-31-10-147.eu-central-1.compute.internal/172.31.10.147:8032
17/05/30 19:38:34 INFO Client: Requesting a new application from cluster with 2 NodeManagers
17/05/30 19:38:34 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (11520 MB per container)
17/05/30 19:38:34 INFO Client: Will allocate AM container, with 11520 MB memory including 1047 MB overhead
17/05/30 19:38:34 INFO Client: Setting up container launch context for our AM
17/05/30 19:38:34 INFO Client: Setting up the launch environment for our AM container
17/05/30 19:38:35 INFO Client: Preparing resources for our AM container
17/05/30 19:38:36 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME.
17/05/30 19:38:39 INFO Client: Uploading resource file:/mnt/tmp/spark-c9b71a05-c3c4-4421-99da-04618b477f69/__spark_libs__9005882140690733981.zip -> hdfs://ip-172-31-10-147.eu-central-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1496172789995_0001/__spark_libs__9005882140690733981.zip
17/05/30 19:38:41 INFO Client: Uploading resource file:/home/hadoop/Main.jar -> hdfs://ip-172-31-10-147.eu-central-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1496172789995_0001/Main.jar
17/05/30 19:38:41 INFO Client: Uploading resource file:/mnt/tmp/spark-c9b71a05-c3c4-4421-99da-04618b477f69/__spark_conf__5018873114441326186.zip -> hdfs://ip-172-31-10-147.eu-central-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1496172789995_0001/__spark_conf__.zip
17/05/30 19:38:41 INFO SecurityManager: Changing view acls to: hadoop
17/05/30 19:38:41 INFO SecurityManager: Changing modify acls to: hadoop
17/05/30 19:38:41 INFO SecurityManager: Changing view acls groups to:
17/05/30 19:38:41 INFO SecurityManager: Changing modify acls groups to:
17/05/30 19:38:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set()
17/05/30 19:38:41 INFO Client: Submitting application application_1496172789995_0001 to ResourceManager
17/05/30 19:38:41 INFO YarnClientImpl: Submitted application application_1496172789995_0001
17/05/30 19:38:42 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:42 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1496173121751
final status: UNDEFINED
tracking URL: http://ip-172-31-10-147.eu-central-1.compute.internal:20888/proxy/application_1496172789995_0001/
user: hadoop
17/05/30 19:38:43 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:44 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:45 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:46 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:47 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:48 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:38:49 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:49 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.31.1.113
ApplicationMaster RPC port: 0
queue: default
start time: 1496173121751
final status: UNDEFINED
tracking URL: http://ip-172-31-10-147.eu-central-1.compute.internal:20888/proxy/application_1496172789995_0001/
user: hadoop
17/05/30 19:38:50 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:51 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:52 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:53 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:54 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:55 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:56 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:57 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:58 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:38:59 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:00 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:01 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:02 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:03 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:04 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:05 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:06 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:07 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:08 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:09 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:10 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:11 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:12 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:13 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:14 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:15 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:16 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:17 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:18 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:19 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:20 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:21 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:22 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:23 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:24 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:24 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1496173121751
final status: UNDEFINED
tracking URL: http://ip-172-31-10-147.eu-central-1.compute.internal:20888/proxy/application_1496172789995_0001/
user: hadoop
17/05/30 19:39:25 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:26 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:27 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:28 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:29 INFO Client: Application report for application_1496172789995_0001 (state: ACCEPTED)
17/05/30 19:39:30 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:30 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.31.0.226
ApplicationMaster RPC port: 0
queue: default
start time: 1496173121751
final status: UNDEFINED
tracking URL: http://ip-172-31-10-147.eu-central-1.compute.internal:20888/proxy/application_1496172789995_0001/
user: hadoop
17/05/30 19:39:31 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:32 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:33 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:34 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:35 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:36 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:37 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:38 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:39 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:40 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:41 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:42 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:43 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:44 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:45 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:46 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:47 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:48 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:49 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:50 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:51 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:52 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:53 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:54 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:55 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:56 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:57 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:58 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:39:59 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:40:00 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:40:01 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:40:02 INFO Client: Application report for application_1496172789995_0001 (state: RUNNING)
17/05/30 19:40:03 INFO Client: Application report for application_1496172789995_0001 (state: FINISHED)
17/05/30 19:40:03 INFO Client:
client token: N/A
diagnostics: N/A
ApplicationMaster host: 172.31.0.226
ApplicationMaster RPC port: 0
queue: default
start time: 1496173121751
final status: FAILED
tracking URL: http://ip-172-31-10-147.eu-central-1.compute.internal:20888/proxy/application_1496172789995_0001/
user: hadoop
17/05/30 19:40:04 INFO Client: Deleted staging directory hdfs://ip-172-31-10-147.eu-central-1.compute.internal:8020/user/hadoop/.sparkStaging/application_1496172789995_0001
Exception in thread "main" org.apache.spark.SparkException: Application application_1496172789995_0001 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1167)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1213)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/05/30 19:40:04 INFO ShutdownHookManager: Shutdown hook called
17/05/30 19:40:04 INFO ShutdownHookManager: Deleting directory /mnt/tmp/spark-c9b71a05-c3c4-4421-99da-04618b477f69
在运行我的spark应用程序时，我使用了以下命令：
spark-submit --class Main.Main --deploy-mode cluster --master yarn --num-executors 2 Main.jar 1.txt 2 10
答案 0 :(得分：0)
这是在EMR上解决的：
spark-submit --deploy-mode cluster --master yarn
请改为使用：
spark-submit --master yarn-client