Question

提前感谢您的帮助。我正在使用提供的Hadoop示例运行一个纱线作业。这项工作永远不会完成并停留在＆＃34;接受＆＃34;州。看看正在打印的内容，似乎工作正在等待完成 - 并且客户不断探索工作状态。

示例作业（来自Hadoop 2.6.0）：

spark-submit --master yarn-client --driver-memory 4g --executor-memory 2g --executor-cores 4  --class org.apache.spark.examples.SparkPi /home/john/spark/spark-1.6.1-bin-hadoop2.6/lib/spark-examples-1.6.1-hadoop2.6.0.jar 100

输出：

....
....
 disabled; ui acls disabled; users with view permissions: Set(john); users with modify permissions: Set(jogn)
16/07/27 17:36:09 INFO yarn.Client: Submitting application 1 to ResourceManager
16/07/27 17:36:09 INFO impl.YarnClientImpl: Submitted application application_1469665943738_0001
16/07/27 17:36:10 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:10 INFO yarn.Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1469666169333
         final status: UNDEFINED
         tracking URL: http://cpt-bdx021:8088/proxy/application_1469665943738_0001/
         user: john
16/07/27 17:36:11 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:12 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:13 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:14 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:15 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:16 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:17 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:18 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:19 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:20 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:21 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
16/07/27 17:36:22 INFO yarn.Client: Application report for application_1469665943738_0001 (state: ACCEPTED)
...........
...........
...........

UPDATE（看起来作业已提交给ResourceManager - 因此＆＃34; ACCEPTED＆＃34;，但ResourceManager＆＃34;看到＆＃34;没有节点或hadoop工作人员实际得到工作）：

$ jps
jps
12404 Jps
12211 NameNode
12315 DataNode
11743 ApplicationHistoryServer
11876 ResourceManager
11542 NodeManager

$ yarn node -list
        16/07/27 23:07:53 INFO client.RMProxy: Connecting to ResourceManager at /192.168.0.5.55:8032
        Total Nodes:0
                 Node-Id             Node-State Node-Http-Address       Number-of-Running-Containers

更新（2）：我使用默认的etc/container-executor.cfg file：

yarn.nodemanager.linux-container-executor.group=#configured value of yarn.nodemanager.linux-container-executor.group
banned.users=#comma separated list of users who can not run applications
min.user.id=1000#Prevent other super-users
allowed.system.users=##comma separated list of system users who CAN run applications

另外，就我而言，我想提一下，我没有hadoop用户或hadoop`用户组。我正在使用我登录系统的默认帐户。如果那很重要。谢谢！

更新（3）：NodeManager日志

org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at 192.168.0.5.55:8031
2016-07-28 00:23:26,083 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Sending out 0 NM container statuses: []
2016-07-28 00:23:26,087 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registering with RM using containers :[]
2016-07-28 00:23:26,233 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMContainerTokenSecretManager: Rolling master-key for container-tokens, got key with id -160570002
2016-07-28 00:23:26,236 INFO org.apache.hadoop.yarn.server.nodemanager.security.NMTokenSecretManagerInNM: Rolling master-key for container-tokens, got key with id -1876215653
2016-07-28 00:23:26,237 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Registered with ResourceManager as 192.168.0.5.55:53034 with total resource of <memory:8192, vCores:8>
2016-07-28 00:23:26,237 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Notifying ContainerManager to unblock new container-requests

Answer 1

你的工作永远不会完成的原因是因为它永远不会进入状态RUNNING（从状态ACCEPTED）。有一个调度程序负责安排哪些应用程序将获得资源，从而声明RUNNING。

有两种可用的调度程序：公平调度程序和容量调度程序。您可以在Hadoop Yarn文档中找到详细信息。如果您可以提供yarn-site.xml，capacity-scheduler.xml和fair-scheduler.xml文件，我会给您更好的帮助:)。

Answer 2

最常见的可能性是您发送作业的队列没有您请求的可用资源。

典型问题可能是：

资源要求（内存和/或核心）。您需要更多可以分配的内存/核心。这可能是因为群集几乎完全使用，或者您的设置不一致。有关this page。
磁盘空间。检查节点空间，有健康检查可能会阻止您运行应用程序。
```
yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage 
```
在多租户/多队列环境中，如果每个队列存在硬资源限制，则您的应用程序可能会遇到这些限制。您可能希望增加设置，或使用更多资源在另一个队列中进行测试。

纱线工作没有过去＆＃34;状态：接受＆＃34;

2 个答案: