从远程客户端

时间：2018-03-13 20:48:39

标签： hadoop apache-spark cluster-computing yarn

我想使用 spark-submit 命令在远程YARN群集上提交Spark作业。我的客户端是Windows机器，集群由主服务器和4个从服务器组成。我将Hadoop配置文件从我的集群复制到远程机器，即 core-site.xml 和 yarn-site.xml ，并在 spark中设置HADOOP_CONF_DIR变量-env.sh 指向他们。

但是，当我使用以下命令提交作业时：

spark-submit --jars hdfs:///user/kmansour/elevation/geotrellis-1.2.1-assembly.jar \  
 --class tutorial.CalculateFlowDirection hdfs:///user/kmansour/elevation/demo_2.11-0.2.0.jar hdfs:///user/kmansour/elevation/TIF/DTM_1m_19_E_17_108_*.tif \  
 --deploy-mode cluster \  
 --master yarn

我忍不住：

INFO yarn.Client: Application report for application_1519070657292_0088 (state: ACCEPTED)

直到我得到这个：

 diagnostics: Application application_1519070657292_0088 failed 2 times due to AM Container for appattempt_1519070657292_0088_000002 exited with  exitCode: 10
    For more detailed output, check application tracking page:http://node1:8088/cluster/app/application_1519070657292_0088Then, click on links to logs of each attempt.
    Diagnostics: Exception from container-launch.
    Container id: container_1519070657292_0088_02_000001
    Exit code: 10
    Stack trace: ExitCodeException exitCode=10:
            at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
            at org.apache.hadoop.util.Shell.run(Shell.java:482)
            at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
            at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
            at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

当我查看应用程序跟踪页面时，我在stderr上得到了这个：

18/03/13 14:48:05 INFO util.SignalUtils: Registered signal handler for TERM
18/03/13 14:48:05 INFO util.SignalUtils: Registered signal handler for HUP
18/03/13 14:48:05 INFO util.SignalUtils: Registered signal handler for INT
18/03/13 14:48:06 INFO yarn.ApplicationMaster: Preparing Local resources
18/03/13 14:48:08 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1519070657292_0088_000002
18/03/13 14:48:08 INFO spark.SecurityManager: Changing view acls to: kmansour
18/03/13 14:48:08 INFO spark.SecurityManager: Changing modify acls to: kmansour
18/03/13 14:48:08 INFO spark.SecurityManager: Changing view acls groups to: 
18/03/13 14:48:08 INFO spark.SecurityManager: Changing modify acls groups to: 
18/03/13 14:48:08 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(kmansour); groups with view permissions: Set(); users  with modify permissions: Set(kmansour); groups with modify permissions: Set()
18/03/13 14:48:08 INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
18/03/13 14:50:15 ERROR yarn.ApplicationMaster: Failed to connect to driver at 132.156.9.98:50687, retrying ...
18/03/13 14:50:15 ERROR yarn.ApplicationMaster: Uncaught exception: 
org.apache.spark.SparkException: Failed to connect to driver!
    at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkDriver(ApplicationMaster.scala:577)
    at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:433)
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:256)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:764)
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:67)
    at org.apache.spark.deploy.SparkHadoopUtil$$anon$2.run(SparkHadoopUtil.scala:66)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:66)
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:762)
    at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:785)
    at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
18/03/13 14:50:15 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: org.apache.spark.SparkException: Failed to connect to driver!)
18/03/13 14:50:16 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: org.apache.spark.SparkException: Failed to connect to driver!)
18/03/13 14:50:16 INFO yarn.ApplicationMaster: Deleting staging directory hdfs://132.156.9.142:8020/user/kmansour/.sparkStaging/application_1519070657292_0088
18/03/13 14:50:16 INFO util.ShutdownHookManager: Shutdown hook called

我的主节点的IP地址是 132.156.9.142 ，我的客户端的IP地址是 132.156.9.98 。该日志显示，当我明确声明 - deploy-mode cluster 时，应用程序主机正在尝试连接到客户端上的驱动程序。

驱动程序驱动程序不应该在群集中的节点上吗？

这是我的配置文件的内容：

spark-defaults.conf ：

spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://132.156.9.142:8020/events
spark.history.fs.logDirectory    hdfs://132.156.9.142:8020/events
spark.serializer                 org.apache.spark.serializer.KryoSerializer
spark.driver.cores               2
spark.driver.memory              5g
spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
spark.executor.instances         4
spark.executor.cores             2
spark.executor.memory            6g
spark.yarn.am.memory             2g
spark.yarn.jars                  hdfs://node1:8020/jars/*.jar

纱-site.xml中：

<configuration>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>node1</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>8192</value>
    </property>
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>1024</value>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>7168</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>2</value>
    </property>
    <property>
        <name>yarn.nodemanager.pmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enabled</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-pmem-ratio</name>
        <value>5</value>
    </property>
</configuration>

core-site.xml ：

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://132.156.9.142:8020</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>C:\Users\kmansour\Documents\hadoop-2.7.4\tmp</value>
    </property>
</configuration>

我对这一切都很新，也许我的推理存在缺陷，任何意见或建议都会有所帮助。

1 个答案:

答案 0 :(得分：2)

您需要更改传递给"no"的订单或参数。在您的配置中：

$ ./bin/scanf_yes_no

program processing....

Do you want to continue (yes/no): what?
error: invalid input.

Do you want to continue (yes/no): obnoxious long line because cat stepped on kbd
error: invalid input.

Do you want to continue (yes/no): yes

program processing....

Do you want to continue (yes/no): yes

program processing....

Do you want to continue (yes/no):
error: invalid input.

Do you want to continue (yes/no): no
that's all folks...

Spark在默认模式下调用（可能是yarn-client），然后你的spark-submit和spark-submit --jars hdfs:///user/kmansour/elevation/geotrellis-1.2.1-assembly.jar \ --class tutorial.CalculateFlowDirection hdfs:///user/kmansour/elevation/demo_2.11-0.2.0.jar hdfs:///user/kmansour/elevation/TIF/DTM_1m_19_E_17_108_*.tif \ --deploy-mode cluster \ --master yarn 作为app参数传递，因为在jar文件位置之后输入了。将其更改为：

--deploy-mode

您将获得真正的纱线群集模式。

相关问题
最新问题