我有一个Flink集群,并用它来运行批处理作业。问题是每项工作完成后,任务管理器的内存仍然很忙。
集群正在docker swarm中运行,并且有两台具有250Gb RAM的机器。
我的flink配置是这样的:
by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
by specifying the --host <hostname> parameter of the bin/jobmanager.sh executable.
# In high availability mode, if you use the bin/start-cluster.sh script and setup
# the conf/masters file, this will be taken care of automatically. Yarn/Mesos
# automatically configure the host name based on the hostname of the node where the
# JobManager runs.
jobmanager.rpc.address: jobmanager17
# The RPC port where the JobManager is reachable.
jobmanager.rpc.port: 6123
# The heap size for the JobManager JVM
jobmanager.heap.mb: 204800
# The heap size for the TaskManager JVM
taskmanager.heap.mb: 204800
# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.
taskmanager.numberOfTaskSlots: 32
# Specify whether TaskManager memory should be allocated when starting up (true) or when
# memory is required in the memory manager (false)
# Important Note: For pure streaming setups, we highly recommend to set this value to `false`
# as the default state backends currently do not use the managed memory.
taskmanager.memory.preallocate: false
# The parallelism used for programs that did not specify and other parallelism.
parallelism.default: 1
集群工作正常,但是它在工作中花费内存,最后使用交换和磁盘结束。
我希望这是一个配置问题,但是如果您需要docker的更多详细信息,请告诉我