Question

我有一个带有三个数据节点（node1，node2和node3）的Hortonworks（HDP 2.4.2）集群。我想检查hortoworks中的节点标记功能。为此，我创建了一个节点标签（x）并将其映射到数据节点node1。所以现在有两个分区：

默认分区 - 包含node2＆amp;节点3
分区x - 包含node1

后来我创建了一个名为“protegrity *并将其映射到节点标签 X 的队列。

现在，当我正在运行任何mapreduce作业时，这个工作正在执行但是在默认分区的“protegrity *队列”中我没想到的。它应该在队列中执行“node1上的protegrity * （标记为分区x）。请参阅附带的调度程序截图。

Scheduler 执行的工作是：hadoop jar ./hadoop-mapreduce/hadoop-mapreduce-examples.jar pi -Dmapred.job.queue.name = protegrity -Dnode_label_expression = x 2 2

capacity_seheduler.xml文件的配置如下所示：

<property>
  <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
  <value>0.2</value>
</property>

<property>
  <name>yarn.scheduler.capacity.maximum-applications</name>
  <value>10000</value>
</property>

<property>
  <name>yarn.scheduler.capacity.node-locality-delay</name>
  <value>40</value>
</property>

<property>
  <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
  <value>false</value>
</property>

<property>
  <name>yarn.scheduler.capacity.resource-calculator</name>
  <value>org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.accessible-node-labels</name>
  <value>x</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.accessible-node-labels.x.capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.accessible-node-labels.x.maximum-capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.acl_administer_queue</name>
  <value>yarn</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
  <value>yarn </value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
  <value>yarn</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.capacity</name>
  <value>50</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.state</name>
  <value>RUNNING</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
  <value>1</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.acl_administer_queue</name>
  <value>yarn</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.acl_submit_applications</name>
  <value>yarn</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.capacity</name>
  <value>20</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.maximum-capacity</name>
  <value>80</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.state</name>
  <value>RUNNING</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.hawqque.user-limit-factor</name>
  <value>2</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.accessible-node-labels</name>
  <value>x</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.accessible-node-labels.x.capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.accessible-node-labels.x.maximum-capacity</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.acl_administer_queue</name>
  <value>yarn </value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.acl_submit_applications</name>
  <value>yarn </value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.capacity</name>
  <value>30</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.maximum-capacity</name>
  <value>60</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.minimum-user-limit-percent</name>
  <value>100</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.ordering-policy</name>
  <value>fifo</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.state</name>
  <value>RUNNING</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.protegrity.user-limit-factor</name>
  <value>1</value>
</property>

<property>
  <name>yarn.scheduler.capacity.root.queues</name>
  <value>default,hawqque,protegrity</value>
</property>

但是当我在 - https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.6/bk_yarn_resource_mgt/content/using_node_labels.html中执行一个不触发mapreduce作业的示例查询时，该作业在标记节点上执行并使用上述队列。

查询是：sudo su yarn hadoop jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -shell_command“sleep 100000”-jar /usr/hdp/current/hadoop-yarn-client/hadoop-yarn-applications-distributedshell.jar -queue protegrity -node_label_expression x

所以，我在这里有点困惑，mapreduce作业的节点标签工作与否!! ?? 如果是，那么我需要一些帮助

Answer 1

你可能现在已经找到了答案，因为这个帖子已经有9个月了，但对于那些还在等待答案的人来说。

节点标记不能与MapReduce作业一起使用，除非我们向队列添加节点标签并使用带有节点标签的队列运行MapReduce作业。根据您的给定配置：队列protegrity将始终使用容量为30％的默认分区，而不是在您标记为x的节点内，因为您需要使您的队列protegrity附加以{{1>}作为默认节点标签。请添加以下配置以使其正常工作：

在队列中提交时，Mapreduce作业未在标记节点上运行，而是在默认分区

1 个答案: