将Capacity Scheduler与Yarn中的节点标签一起使用时,无法获取队列指标

时间:2019-10-07 10:49:22

标签: hadoop yarn hadoop3

我有一个带有两个队列的容量调度程序设置-“低优先级”,“常规优先级”。 有两个节点标签“ low”和“ regular”。 低优先级队列对“低”节点标签具有100%的访问权限,而常规优先级队列对“常规”节点标签具有100%的访问权限。

容量调度程序配置(capacity-scheduler.xml):

<configuration>

<property>
    <name>yarn.scheduler.capacity.maximum-applications</name>
    <value>1000</value>
</property>

<property>
    <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
    <value>0.2</value>
</property>

<property>
    <name>yarn.scheduler.capacity.resource-calculator</name>
    <value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
    <description>
        The ResourceCalculator implementation to be used to compare
        Resources in the scheduler.
        The default i.e. DefaultResourceCalculator only uses Memory while
        DominantResourceCalculator uses dominant-resource to compare
        multi-dimensional resources such as Memory, CPU etc.
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.root.queues</name>
    <value>low-priority,regular-priority</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels</name>
    <value>*</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels.regular.capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels.regular.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels.low.capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.accessible-node-labels.low.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.default.state</name>
    <value>RUNNING</value>
    <description>
        The state of the default queue. State can be one of RUNNING or STOPPED.
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
    <value>*</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
    <value>*</value>
    <description>
        The ACL of who can administer jobs on the default queue.
    </description>
</property>

<property>
    <name>yarn.scheduler.capacity.node-locality-delay</name>
    <value>40</value>
</property>

<property>
    <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
    <value>false</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.capacity</name>
    <value>50</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.ordering-policy</name>
    <value>fair</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.accessible-node-labels</name>
    <value>low</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.default-node-label-expression</name>
    <value>low</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.accessible-node-labels.low.capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.accessible-node-labels.low.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.default.state</name>
    <value>RUNNING</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.capacity</name>
    <value>50</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.ordering-policy</name>
    <value>fair</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.accessible-node-labels</name>
    <value>regular</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.default-node-label-expression</name>
    <value>regular</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.accessible-node-labels.regular.capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.regular-priority.accessible-node-labels.regular.maximum-capacity</name>
    <value>100</value>
</property>

<property>
    <name>yarn.scheduler.capacity.root.low-priority.default.state</name>
    <value>RUNNING</value>
</property>

附带了纱线ui容量调度程序配置。 yarn ui scheduler 当我在(rm-ip:port / jmx)中看到队列“低优先级”和“常规优先级”发出的QueueMetrics时,它会显示availableMB和availableVCores的正确值,但是当我向任何队列提交作业时,是在jmx指标(例如,未决MB,未决Vcore,可用MB,可用VCore等)上没有更新。仅AppsRunning,ActiveApplications等正在更新。 无法找到提交作业后为何指标未更新的原因。

仅当启用了节点标签时,该问题才会出现。当禁用node-label且仅使用队列时,一切正常。

0 个答案:

没有答案