我有一个带有两个队列的容量调度程序设置-“低优先级”,“常规优先级”。 有两个节点标签“ low”和“ regular”。 低优先级队列对“低”节点标签具有100%的访问权限,而常规优先级队列对“常规”节点标签具有100%的访问权限。
容量调度程序配置(capacity-scheduler.xml):
<configuration>
<property>
<name>yarn.scheduler.capacity.maximum-applications</name>
<value>1000</value>
</property>
<property>
<name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
<value>0.2</value>
</property>
<property>
<name>yarn.scheduler.capacity.resource-calculator</name>
<value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
<description>
The ResourceCalculator implementation to be used to compare
Resources in the scheduler.
The default i.e. DefaultResourceCalculator only uses Memory while
DominantResourceCalculator uses dominant-resource to compare
multi-dimensional resources such as Memory, CPU etc.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.queues</name>
<value>low-priority,regular-priority</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.accessible-node-labels</name>
<value>*</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.accessible-node-labels.regular.capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.accessible-node-labels.regular.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.accessible-node-labels.low.capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.accessible-node-labels.low.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.state</name>
<value>RUNNING</value>
<description>
The state of the default queue. State can be one of RUNNING or STOPPED.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
<value>*</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
<value>*</value>
<description>
The ACL of who can administer jobs on the default queue.
</description>
</property>
<property>
<name>yarn.scheduler.capacity.node-locality-delay</name>
<value>40</value>
</property>
<property>
<name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
<value>false</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.low-priority.capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.low-priority.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.low-priority.ordering-policy</name>
<value>fair</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.low-priority.accessible-node-labels</name>
<value>low</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.low-priority.default-node-label-expression</name>
<value>low</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.low-priority.accessible-node-labels.low.capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.low-priority.accessible-node-labels.low.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.low-priority.default.state</name>
<value>RUNNING</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.regular-priority.capacity</name>
<value>50</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.regular-priority.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.regular-priority.ordering-policy</name>
<value>fair</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.regular-priority.accessible-node-labels</name>
<value>regular</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.regular-priority.default-node-label-expression</name>
<value>regular</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.regular-priority.accessible-node-labels.regular.capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.regular-priority.accessible-node-labels.regular.maximum-capacity</name>
<value>100</value>
</property>
<property>
<name>yarn.scheduler.capacity.root.low-priority.default.state</name>
<value>RUNNING</value>
</property>
附带了纱线ui容量调度程序配置。 yarn ui scheduler 当我在(rm-ip:port / jmx)中看到队列“低优先级”和“常规优先级”发出的QueueMetrics时,它会显示availableMB和availableVCores的正确值,但是当我向任何队列提交作业时,是在jmx指标(例如,未决MB,未决Vcore,可用MB,可用VCore等)上没有更新。仅AppsRunning,ActiveApplications等正在更新。 无法找到提交作业后为何指标未更新的原因。
仅当启用了节点标签时,该问题才会出现。当禁用node-label且仅使用队列时,一切正常。