Question

我们有一个配置了Cloudera CDH 5.4.2分发的hadoop集群，配有Yarn公平调度程序，用于调度和管理资源。我们有18个工作节点，总计963GB主内存和288个vcores。

现在的问题是，无论应用程序的资源需求有多小，我一次只能运行一个应用程序。例如，这里是动态资源池状态：

状态 YARN正在使用288个vcores和942 GiB的内存。资源池使用

此表格和右侧的图表仅包含YARN的指标。资源池名称分配的内存分配的VCores分配的容器待定容器
root 0 B 0 0 0

user1 0 B 0 0 1
默认0 B 0 0 0
user2 22 GiB 11 11 1

对于用户2，即使在288个vcores和942 GiB内存中只分配了22GB和11个vcores，仍然有一个待处理的容器！知道发生了什么事吗？前几天不是这种情况，不确定是什么设置造成这种情况？

这是我的yarn-site.xml内容

 <?xml version="1.0" encoding="UTF-8"?>

    <!--Autogenerated by Cloudera Manager-->
    <configuration>
      <property>
        <name>yarn.acl.enable</name>
        <value>true</value>
      </property>
      <property>
        <name>yarn.admin.acl</name>
        <value>*</value>
      </property>
      <property>
        <name>yarn.resourcemanager.address</name>
        <value>hadoopmaster:8032</value>
      </property>
      <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>hadoopmaster:8033</value>
      </property>
      <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>hadoopmaster:8030</value>
      </property>
      <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>hadoopmaster:8031</value>
      </property>
      <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>hadoopmaster:8088</value>
      </property>
      <property>
        <name>yarn.resourcemanager.webapp.https.address</name>
        <value>hadoopmaster:8090</value>
      </property>
      <property>
        <name>yarn.resourcemanager.client.thread-count</name>
        <value>50</value>
      </property>
      <property>
        <name>yarn.resourcemanager.scheduler.client.thread-count</name>
        <value>50</value>
      </property>
      <property>
        <name>yarn.resourcemanager.admin.client.thread-count</name>
        <value>1</value>
      </property>
      <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>2048</value>
      </property>
      <property>
        <name>yarn.scheduler.increment-allocation-mb</name>
        <value>512</value>
      </property>
      <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>65536</value>
      </property>
      <property>
        <name>yarn.scheduler.minimum-allocation-vcores</name>
        <value>1</value>
      </property>
      <property>
        <name>yarn.scheduler.increment-allocation-vcores</name>
        <value>1</value>
      </property>
      <property>
        <name>yarn.scheduler.maximum-allocation-vcores</name>
        <value>24</value>
      </property>
      <property>
        <name>yarn.resourcemanager.amliveliness-monitor.interval-ms</name>
        <value>1000</value>
      </property>
      <property>
        <name>yarn.am.liveness-monitor.expiry-interval-ms</name>
        <value>600000</value>
      </property>
      <property>
        <name>yarn.resourcemanager.am.max-attempts</name>
        <value>2</value>
      </property>
      <property>
        <name>yarn.resourcemanager.container.liveness-monitor.interval-ms</name>
        <value>600000</value>
      </property>
      <property>
        <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name>
        <value>1000</value>
      </property>
      <property>
        <name>yarn.nm.liveness-monitor.expiry-interval-ms</name>
        <value>600000</value>
      </property>
      <property>
        <name>yarn.resourcemanager.resource-tracker.client.thread-count</name>
        <value>50</value>
      </property>
      <property>
        <name>yarn.application.classpath</name>
        <value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*</value>
      </property>
      <property>
        <name>yarn.resourcemanager.scheduler.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
      </property>
      <property>
        <name>yarn.scheduler.fair.user-as-default-queue</name>
        <value>true</value>
      </property>
      <property>
        <name>yarn.scheduler.fair.preemption</name>
        <value>true</value>
      </property>
      <property>
        <name>yarn.scheduler.fair.sizebasedweight</name>
        <value>false</value>
      </property>
      <property>
        <name>yarn.scheduler.fair.assignmultiple</name>
        <value>true</value>
      </property>
      <property>
        <name>yarn.resourcemanager.max-completed-applications</name>
        <value>10000</value>
      </property>
    </configuration>

Answer 1

请查看fair-scheduler.xml文件中的值＆＃34; maxRunningApps＆＃34;它定义了可以在任何队列中运行的并发应用程序的数量。

请参阅https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/FairScheduler.html

Apache Yarn公平调度程序一次只允许一个应用程序

1 个答案: