Question

我想在 Hadoop YARN 群集模式下运行我的spark Job，我使用以下命令：

spark-submit --master yarn-cluster 
             --driver-memory 1g 
             --executor-memory 1g
             --executor-cores 1 
             --class com.dc.analysis.jobs.AggregationJob
               sparkanalitic.jar param1 param2 param3

我收到错误，请说明错误是什么，命令是否正确。我正在使用CDH 5.3.1。

Diagnostics: Application application_1424284032717_0066 failed 2 times due 
to AM Container for appattempt_1424284032717_0066_000002 exited with  
exitCode: 15 due to: Exception from container-launch.

Container id: container_1424284032717_0066_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
    at org.apache.hadoop.util.Shell.run(Shell.java:455)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:197)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)  

Container exited with a non-zero exit code 15
.Failing this attempt.. Failing the application.
     ApplicationMaster host: N/A
     ApplicationMaster RPC port: -1
     queue: root.hdfs
     start time: 1424699723648
     final status: FAILED
     tracking URL: http://myhostname:8088/cluster/app/application_1424284032717_0066
     user: hdfs

2015-02-23 19:26:04 DEBUG Client - stopping client from cache: org.apache.hadoop.ipc.Client@4085f1ac
2015-02-23 19:26:04 DEBUG Utils - Shutdown hook called
2015-02-23 19:26:05 DEBUG Utils - Shutdown hook called

非常感谢任何帮助。

Answer 1

这可能意味着很多事情，对于我们来说，由于不支持的Java类版本，我们得到类似的错误消息，我们通过删除项目中引用的Java类来解决问题。

使用此命令查看详细的错误消息：

yarn logs -applicationId application_1424284032717_0066

Answer 2

你应该删除＆＃34; .setMaster（＆＃34; local＆＃34;）＆＃34;在代码中。

Answer 3

该命令看起来正确。

我遇到的是＆＃34;退出代码15＆＃34;通常表示TableNotFound异常。这通常意味着您提交的代码中存在错误。

您可以访问跟踪网址进行检查。

Answer 4

通过将<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min.js"></script> <form role="form" class="form-inline"> <div class="form-group"> <label class="control-label" for="Q1Month"> Quarter 1 Month </label> <select class="form-control" id="Q1Month"> <option selected value=''>--Select Month--</option> <option value='1'>January</option> <option value='2'>February</option> <option value='3'>March</option> <option value='4'>April</option> <option value='5'>May</option> <option value='6'>June</option> <option value='7'>July</option> <option value='8'>August</option> <option value='9'>September</option> <option value='10'>October</option> <option value='11'>November</option> <option value='12'>December</option> </select> </div> </form>放在hive-site.xml目录中解决退出代码问题。

Answer 5

删除第"spark.master":"local[*]行＆＃34;如果您正在群集下运行spark作业，请在spark配置文件中。

假设在本地电脑上运行，请包含它。

摩尼

问题在Yarn Cluster上运行Spark Job

5 个答案: