我有一个简单的火花应用程序用于学习puprose ...这个scala程序parallelizr数据列表并将RDD写入Hadoop中的文件。
import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
object HelloSpark {
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("HelloSPark1").setMaster(args(0))
val sc = new SparkContext(conf)
val i = List(1,4,2,11,23,45,67,8,909,5,1,8,"agarwal",19,11,12,34,8031,"aditya")
val b = sc.parallelize(i,3)
b.saveAsTextFile(args(1))
}
}
i create a jar file and when i run it on my cluster it throws error when i run it as --master YARN and --deploy-mode cluster using following command
spark-submit --class "HelloSpark" --master yarn --deploy-mode cluster sparkappl_2.11-1.0.jar yarn /user
/letsbigdata9356/sparktest/run6
client token: N/A
diagnostics: Application application_1483332319047_3791 failed 2 times due to AM Container for appattempt_1483332319047_3791_000002 e
xited with exitCode: 15
For more detailed output, check application tracking page:http://a.cloudxlab.com:8088/cluster/app/application_1483332319047_3791Then, click on
links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e77_1483332319047_3791_02_000001
Exit code: 15
Stack trace: ExitCodeException exitCode=15:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
at org.apache.hadoop.util.Shell.run(Shell.java:487)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 15
Failing this attempt. Failing the application.
Container exited with a non-zero exit code 15
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1484231621733
final status: FAILED
tracking URL: http://a.cloudxlab.com:8088/cluster/app/application_1483332319047_3791
user: letsbigdata9356
Exception in thread "main" org.apache.spark.SparkException: Application application_1483332319047_3791 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:974)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1020)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:685)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
17/01/12 14:34:10 INFO ShutdownHookManager: Shutdown hook called
但是当我在纱线客户端模式或本地模式下使用以下命令运行它时，它可以正常工作
spark-submit --class "HelloSpark" sparkappl_2.11-1.0.jar yarn-client /user/letsbigdata9356/sparktest/run
5
or
spark-submit --class "HelloSpark" sparkappl_2.11-1.0.jar local /user/letsbigdata9356/sparktest/run
7
我是新手，请你帮我解决和了解这个问题。