我遇到了sparkSubmit作业的问题。
我尝试通过sparkSubmit在hadoop上运行一个作业： （我评论了所有的代码，以确保它不是来自工作。我当然也尝试了所有的代码，我有同样的问题）
public static void main( String[] args )
{
LOGGER.info("START EXTRACT");
LOGGER.info("args: {}", args);
//TODO delete after tests
LOGGER.info("END EXTRACT");
System.exit(0);
/* real code in commentary */
}
当我开始工作时，我有这个：
#10 Mar 2017 11:38:42,128 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:42 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:42 INFO Client:
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: client token: N/A
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: diagnostics: N/A
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: ApplicationMaster host: N/A
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: ApplicationMaster RPC port: -1
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: queue: default
#10 Mar 2017 11:38:42,131 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: start time: 1489142321100
#10 Mar 2017 11:38:42,132 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: final status: UNDEFINED
#10 Mar 2017 11:38:42,132 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: tracking URL: http:***************/proxy/application_1460635834146_0167/
#10 Mar 2017 11:38:42,132 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: user: Joe
#10 Mar 2017 11:38:43,135 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:43 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:44,138 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:44 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:45,140 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:45 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:46,144 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:46 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:47,146 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:47 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:48,148 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:48 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:49,150 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:49 INFO Client: Application report for application_1460635834146_0167 (state: ACCEPTED)
#10 Mar 2017 11:38:50,153 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:50 INFO Client: Application report for application_1460635834146_0167 (state: FAILED)
#10 Mar 2017 11:38:50,156 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: 17/03/10 11:38:50 INFO Client:
#10 Mar 2017 11:38:50,156 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: client token: N/A
#10 Mar 2017 11:38:50,156 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: diagnostics: Application application_1460635834146_0167 failed 2 times due to AM Container for appattempt_1460635834146_0167_000002 exited with exitCode: 0
#10 Mar 2017 11:38:50,156 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: For more detailed output, check application tracking page:http://**************/cluster/app/application_1460635834146_0167Then, click on links to logs of each attempt.
#10 Mar 2017 11:38:50,156 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: Diagnostics: Failing this attempt. Failing the application.
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: ApplicationMaster host: N/A
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: ApplicationMaster RPC port: -1
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: queue: default
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: start time: 1489142321100
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: final status: FAILED
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: tracking URL: http://*******************/cluster/app/application_1460635834146_0167
#10 Mar 2017 11:38:50,157 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: user: Joe
#10 Mar 2017 11:38:50,158 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: Error: application failed with exception
#10 Mar 2017 11:38:50,159 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: org.apache.spark.SparkException: Application finished with failed status
#10 Mar 2017 11:38:50,159 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.yarn.Client.run(Client.scala:622)
#10 Mar 2017 11:38:50,159 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.yarn.Client$.main(Client.scala:647)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.yarn.Client.main(Client.scala)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at java.lang.reflect.Method.invoke(Method.java:497)
#10 Mar 2017 11:38:50,160 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:577)
#10 Mar 2017 11:38:50,161 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:174)
#10 Mar 2017 11:38:50,161 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:197)
#10 Mar 2017 11:38:50,161 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112)
#10 Mar 2017 11:38:50,161 [DEBUG] (InputStreamReaderRunnable.java:run:32): Stream spark: at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
#10 Mar 2017 11:38:50,174 [DEBUG] (SparkServicesImpl.java:submitSparkJobs:124): End submit SparkJob. Exit value: 1
#10 Mar 2017 11:38:50,174 [DEBUG] (Main.java:main:42): END sparkTest
当我检查登录hadoop时，我有这个：
17/03/10 11:07:56 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
17/03/10 11:07:57 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1460635834146_0165_000001
17/03/10 11:07:57 INFO spark.SecurityManager: Changing view acls to: yarn,Joe
17/03/10 11:07:57 INFO spark.SecurityManager: Changing modify acls to: yarn,Joe
17/03/10 11:07:57 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, Joe); users with modify permissions: Set(yarn, Joe)
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Waiting for spark context initialization
17/03/10 11:07:57 INFO jobextractticket.ExtractLauncher: START EXTRACT
17/03/10 11:07:57 INFO jobextractticket.ExtractLauncher: args: telrptextract-application-context.xml
17/03/10 11:07:57 INFO jobextractticket.ExtractLauncher: END EXTRACT
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before final status was reported.)
17/03/10 11:07:57 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1460635834146_0165
End of LogType:stderr
我不明白为什么我的作业有0输出代码，但是sparkSubmit的错误代码。
我尝试了完整的工作，没有成功，然后在工作中使用尽可能少的代码（如上面的代码部分所示） 如果有人有想法:)谢谢！