在客户端模式下远程连接到纱线群集上的火花

时间:2015-10-27 18:44:15

标签: hadoop apache-spark rstudio yarn sparkr

我在纱线群集上有一个远程火花,如果我使用该群集上托管的rstudio服务器(网络版)以客户端模式连接,我可以执行以下操作:

sc <- SparkR::sparkR.init(master = "yarn-client") 

但是,如果我尝试在本地计算机上使用rstudio以同样的方式连接到该spark群集,那么我就会出错:

ERROR SparkContext: Error initializing SparkContext.
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master
...
ERROR Utils: Uncaught exception in thread nioEventLoopGroup-2-2
java.lang.NullPointerException
...
ERROR RBackendHandler: createSparkContext on org.apache.spark.api.r.RRDD failed
Error in invokeJava(isStatic = TRUE, className, methodName, ...) : 
org.apache.spark.SparkException: Yarn application has already ended! It might have been killed or unable to launch application master.

hadoop应用程序跟踪页面上的更详细的错误消息如下:

User:   blueivy
Name:   SparkR
Application Type:   SPARK
Application Tags:   
State:  FAILED
FinalStatus:    FAILED
Started:    27-Oct-2015 11:07:09
Elapsed:    4mins, 39sec
Tracking URL:   History
Diagnostics:    
Application application_1445628650748_0027 failed 2 times due to AM Container for appattempt_1445628650748_0027_000002 exited with exitCode: 10
For more detailed output, check application tracking page:http://master:8088/proxy/application_1445628650748_0027/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1445628650748_0027_02_000001
Exit code: 10
Stack trace: ExitCodeException exitCode=10:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618)
at java.lang.Thread.run(Thread.java:785)
Container exited with a non-zero exit code 10
Failing this attempt. Failing the application. 

我有与远程集群的hadoop和spark相同的配置和环境:spark 1.5.1,hadoop 2.6.0和ubuntu 14.04。任何人都可以帮我找到我的错误吗?

0 个答案:

没有答案