使Yarn动态分配Spark资源

Question

我有一个由Yarn管理的集群并运行Spark作业，这些组件是使用Ambari（2.6.3.0-235）安装的。我有6个主机，每个主机有6个核心。我使用Fair Scheduler

我希望Yarn自动添加/删除执行程序核心，但是无论我做什么都行不通

相关的Spark配置（在Ambari中配置）：

spark.dynamicAllocation.schedulerBacklogTimeout 10s
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout 5s
spark.driver.memory 4G
spark.dynamicAllocation.enabled true
spark.dynamicAllocation.initialExecutors 6 (has no effect - starts with 2)
spark.dynamicAllocation.maxExecutors 10
spark.dynamicAllocation.minExecutors 1
spark.scheduler.mode FAIR
spark.shuffle.service.enabled true
SPARK_EXECUTOR_MEMORY="36G"

相关纱线配置（在Ambari中配置）：

yarn.nodemanager.aux-services mapreduce_shuffle,spark_shuffle,spark2_shuffle
YARN Java heap size 4096
yarn.resourcemanager.scheduler.class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
yarn.scheduler.fair.preemption true
yarn.nodemanager.aux-services.spark2_shuffle.class org.apache.spark.network.yarn.YarnShuffleService 
yarn.nodemanager.aux-services.spark2_shuffle.classpath {{stack_root}}/${hdp.version}/spark2/aux/*
yarn.nodemanager.aux-services.spark_shuffle.class org.apache.spark.network.yarn.YarnShuffleService
yarn.nodemanager.aux-services.spark_shuffle.classpath {{stack_root}}/${hdp.version}/spark/aux/*
Minimum Container Size (VCores) 0
Maximum Container Size (VCores) 12 
Number of virtual cores 12

我还遵循Dynamic resource allocation，并通过了配置外部随机播放服务的所有步骤，我复制了yarn-shuffle jar：

cp /usr/hdp/2.6.3.0-235/spark/aux/spark-2.2.0.2.6.3.0-235-yarn-shuffle.jar /usr/hdp/2.6.3.0-235/hadoop-yarn/lib/

我看到只有3个内核分配给应用程序（默认执行程序是2个，所以我猜它的驱动程序是2个）：

尽管有许多任务待处理：