如何配置Flink群集内存管理

时间:2019-06-11 11:30:22

标签: apache-flink docker-swarm

我有一个Flink集群,并用它来运行批处理作业。问题是每项工作完成后,任务管理器的内存仍然很忙。

集群正在docker swarm中运行,并且有两台具有250Gb RAM的机器。

我的flink配置是这样的:


by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.



by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
# In high availability mode, if you use the bin/start-cluster.sh script and setup
# the conf/masters file, this will be taken care of automatically. Yarn/Mesos
# automatically configure the host name based on the hostname of the node where the
# JobManager runs.

jobmanager.rpc.address: jobmanager17

# The RPC port where the JobManager is reachable.

jobmanager.rpc.port: 6123


# The heap size for the JobManager JVM

jobmanager.heap.mb: 204800


# The heap size for the TaskManager JVM

taskmanager.heap.mb: 204800


# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.

taskmanager.numberOfTaskSlots: 32

# Specify whether TaskManager memory should be allocated when starting up (true) or when
# memory is required in the memory manager (false)
# Important Note: For pure streaming setups, we highly recommend to set this value to `false`
# as the default state backends currently do not use the managed memory.

taskmanager.memory.preallocate: false

# The parallelism used for programs that did not specify and other parallelism.

parallelism.default: 1


集群工作正常,但是它在工作中花费内存,最后使用交换和磁盘结束。

我希望这是一个配置问题,但是如果您需要docker的更多详细信息,请告诉我

0 个答案:

没有答案