运行跟随命令以将作业提交到纱线,但是不希望分配任务管理器的数量。群集具有足够的资源<220vcores,720G mem>
/data/clusterserver/flink-1.7.2/bin/flink run -m yarn-cluster -yn 4 -ys 2 /data/clusterserver/flink-1.7.2/examples/batch/WordCount.jar --input hdfs:///tmp/wordcount.txt --output hdfs:///tmp/count.result
2019-04-10 15:04:57,397 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-04-10 15:04:57,397 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2019-04-10 15:04:57,405 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - The argument yn is deprecated in will be ignored.
2019-04-10 15:04:57,405 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli - The argument yn is deprecated in will be ignored.
2019-04-10 15:04:57,554 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Cluster specification: ClusterSpecification{masterMemoryMB=2048, taskManagerMemoryMB=2048, numberTaskManagers=4, slotsPerTaskManager=2}
2019-04-10 15:04:58,017 WARN org.apache.flink.yarn.AbstractYarnClusterDescriptor - The configuration directory ('/data/clusterserver/flink-1.7.2/conf') contains both LOG4J and Logback configuration files. Please delete or rename one of them.
2019-04-10 15:05:01,419 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting application master application_1554348641734_0105
2019-04-10 15:05:01,446 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1554348641734_0105
2019-04-10 15:05:01,447 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Waiting for the cluster to be allocated
2019-04-10 15:05:01,450 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying cluster, current state ACCEPTED
2019-04-10 15:05:05,747 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN application has been deployed successfully.
Starting execution of program
从INFO日志中可以看到,发送到yarn的参数是
{masterMemoryMB=2048, taskManagerMemoryMB=2048, numberTaskManagers=4, slotsPerTaskManager=2}
,但是flink WebUI中只有一个taskManager。
使用
yarn-session.sh
和-n
也遇到相同的问题,即从命令行控制taskManager的数量不起作用。