申请任务管理器时,为什么“ -n”或“ -yn”之类的参数不起作用

时间:2019-04-10 04:02:50

标签: apache-flink

运行跟随命令以将作业提交到纱线,但是不希望分配任务管理器的数量。群集具有足够的资源<220vcores,720G mem>

/data/clusterserver/flink-1.7.2/bin/flink run -m yarn-cluster -yn 4 -ys 2  /data/clusterserver/flink-1.7.2/examples/batch/WordCount.jar --input hdfs:///tmp/wordcount.txt --output hdfs:///tmp/count.result
2019-04-10 15:04:57,397 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-04-10 15:04:57,397 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-04-10 15:04:57,405 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument yn is deprecated in will be ignored.
2019-04-10 15:04:57,405 INFO  org.apache.flink.yarn.cli.FlinkYarnSessionCli                 - The argument yn is deprecated in will be ignored.
2019-04-10 15:04:57,554 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Cluster specification: ClusterSpecification{masterMemoryMB=2048, taskManagerMemoryMB=2048, numberTaskManagers=4, slotsPerTaskManager=2}
2019-04-10 15:04:58,017 WARN  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - The configuration directory ('/data/clusterserver/flink-1.7.2/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2019-04-10 15:05:01,419 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Submitting application master application_1554348641734_0105
2019-04-10 15:05:01,446 INFO  org.apache.hadoop.yarn.client.api.impl.YarnClientImpl         - Submitted application application_1554348641734_0105
2019-04-10 15:05:01,447 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Waiting for the cluster to be allocated
2019-04-10 15:05:01,450 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - Deploying cluster, current state ACCEPTED
2019-04-10 15:05:05,747 INFO  org.apache.flink.yarn.AbstractYarnClusterDescriptor           - YARN application has been deployed successfully.
Starting execution of program

从INFO日志中可以看到,发送到yarn的参数是  {masterMemoryMB=2048, taskManagerMemoryMB=2048, numberTaskManagers=4, slotsPerTaskManager=2},但是flink WebUI中只有一个taskManager。

使用  yarn-session.sh-n也遇到相同的问题,即从命令行控制taskManager的数量不起作用。

  • 链接版本1.7.2
  • hadoop版本2.8.5(容量计划程序)
  • java:1.8.0_91

1 个答案:

答案 0 :(得分:2)

从Flink 1.5开始,Flink支持完全的资源弹性。这意味着Flink在Yarn上运行时能够动态启动和停止TaskManagers。结果是,您不再需要为Yarn上的Flink集群指定容器数/ TaskManagers。这就是为什么Flink不再尊重-n / -yn选项的原因。

相反,Flink考虑了作业的并行性,以计算执行该作业所需的插槽数,然后启动满足插槽要求所需的TaskManagers

您可以找到有关部署更改herehere的更多信息。