Question

给出一个用memory limit启动的Docker容器，我想知道是否有可能检测到何时执行了该内存限制，因此该容器将因为内存不足而崩溃。是否有一些我可以捕获的信号？或者一些我可以轮询的日志文件？事后检测到这一点，即使有几分钟的延迟仍然会有用

Answer 1

您可以根据以下说明来编写脚本：

# for i in $(docker ps --no-trunc | awk '{print $1}' | grep -v CONTAINER); do \
LIMIT=`cat /sys/fs/cgroup/memory/docker/${i}/memory.limit_in_bytes`; \
USAGE=`systemd-cgtop -b --iterations 1 | grep ${i} | awk '{print $4}'`; \
echo Container ${i} is using $USAGE and its limited at $LIMIT; done;

Container e831b58193287ac61038da29f488423da894d44f810a9b2ff6df63b800d98217 is using 4.5G and its limited at 9223372036854771712
Container 29b484032c895b84e7297091b0b0fc69513c34a03e10ac7906e428538a47e1a1 is using 19.4M and its limited at 9223372036854771712
Container a6ff3e99790f7d3db1da14736e4f48623512fbd46b2683a158aed1736c17073d is using 14.7M and its limited at 9223372036854771712
Container 36db60a74eab475397371bfa0d1ea432486841d036c7cc54e527df1f7b95bbe9 is using 15.3M and its limited at 9223372036854771712
Container 9be4c12074f05eb00e4e5d17eafe733651ff7b7e962faf328e4dd8a10ca8fee7 is using 216.0K and its limited at 9223372036854771712
Container e9da54900df72ab4b1c6233f51b011a68246315d714278e11b54e7b18367593c is using 2.6G and its limited at 9223372036854771712
Container a670124330c56af011d9bc9b86cbd5571c45af9d2d2dbeffdc6d2aee0b04909f is using 350.1M and its limited at 9223372036854771712
Container e2688ff5493c4517ae55e11281ba07c2743f2464da3c08f88fdce1f4bd0c2d8d is using 2.6G and its limited at 9223372036854771712
Container b0b92b056f82f9b4bff5462b6b896d366245dc9af9847d40f85d20b0497c5601 is using 1.1G and its limited at 9223372036854771712
Container 41e9c719c65e5bc568596f4046fab1acd0ac0e15c93c76563ef25ef5c483b357 is using 61.7M and its limited at 9223372036854771712
Container 1ec5305130846edb67218b2010306090d1efed6e6bc12d04de68fae94db330bd is using 155.1M and its limited at 9223372036854771712
Container 1d3cf6ed42eae789ecc01503e704ff29aa4bd9adb0764baf28f393a882bc0c42 is using 366.3M and its limited at 9223372036854771712
Container 4e96919f2a0e5d969ee1907bffe45fed34b7a4e3dbb7a29ab18169579aed7994 is using 68.7M and its limited at 9223372036854771712
Container 31cc3d3a04435c49d7afb8eda5bab0ef0ee309e1dda531fa8e90a012a5543e90 is using 19.0M and its limited at 9223372036854771712
Container 64dd6daaebc1ef852dcb3a0d2294b52c52d92889ab3f6749a99e45cf89eaa7a6 is using 13.0M and its limited at 9223372036854771712
Container 4f33db06c957306ea19a14c22d5248a161fe5d5344c9d4136eb34c13eb76664b is using 14.7M and its limited at 9223372036854771712
Container 85d9a633adc138f6f506df77ec0828078f521fc5cdfa3cc85e20ab43f3ec0979 is using 40.7M and its limited at 9223372036854771712
Container c102bc6691955d1c732a725c3e9b131c3a07c507996076fb64fce3586a46dbb0 is using 114.2M and its limited at 9223372036854771712

说明：

您会收到很长的容器ID
您将获得每个Docker容器的cgroup memory.limit_in_bytes
您使用systemd-cgtop实用程序来获取当前使用情况
您必须进行一些单位大小计算（因为systemd-cgtop返回G / M / K / B和memory.limit_in_bytes始终为字节）
按使用量划分使用量，配置阈值（即0.8）并捕获您正在使用的任何监视系统

我不限制容器的内存，因此通常将9223372036854771712视为LIMIT。就您而言，它将以字节为单位。

正如其他用户回答的那样，您还可以使用Docker实用程序（不确定正在运行哪个版本），这会容易得多：

docker stats --no-stream | tr -d "%" | awk '{if($7>10)print "Warning: " $1 "/" $2" is consuming more than 80% of its availabile memory ("$7"%)"}'

Answer 2

一种监视此情况的方法是使用 docker stats 命令。您甚至可以在实施限制之前监视容器的使用情况。

https://docs.docker.com/config/containers/runmetrics/#docker-stats

docker stats container1 container2

您也可以执行一些bash脚本：

how to find MAX memory from docker stats?

另一种选择是查看 docker事件：

https://www.systutorials.com/docs/linux/man/1-docker-events/

Docker容器将报告以下事件：

attach, commit, copy, create, destroy, detach, die, exec_create, exec_detach, exec_start, export, kill, oom, pause, rename, resize, restart, start, stop, top, unpause, update

Answer 3

如果docker由于内存限制而杀死了您的容器，您将看到：

docker inspect --format '{{.State.OOMKilled}}' ${container_id}

设置为true。

请注意，如果操作系统在达到docker内存限制之前杀死了容器中运行的进程，则您看到的只是您的应用收到的SIGKILL。

您可以主动docker stats监视容器，以查看其是否接近极限，如此处其他答案所述。

并且可以几乎实时地监视docker events，以查看是否有任何因OOM而被杀死的容器：

docker events --filter type=container --filter event=oom

您可以调整上述事件命令，以监视特定的容器或受时间限制。有关可用的标志，请参见documentation。

如何检测Docker容器何时达到其配置的内存限制

3 个答案: