Docker Container退出137

时间:2019-09-05 17:01:43

标签: docker docker-compose raspbian

我在树莓派上运行了多个docker容器。不久之后,其中两个容器以137退出。

137是OOM(内存不足)或Sigterm。据我所知,我没有发送Sigterm。我用docker stats记录了内存使用情况,并且该内存也在范围之内:

CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
646046130d91        app_gui      0.01%               1.828MiB / 3.814GiB   0.05%               233kB / 229kB       5.14MB / 4.1kB      2
36a212620e76        app_core     78.39%              190.7MiB / 3.814GiB   4.88%               542kB / 448kB       53.7MB / 4.1kB      24
8b1cb459328d        mongo               0.80%               30.04MiB / 3.814GiB   0.77%               332kB / 452kB       61.9MB / 15.7MB     21
dfe65081e256        watchtower          0.00%               5.105MiB / 3.814GiB   0.13%               22kB / 1.18kB       9.49MB / 0B         10
CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
646046130d91        app_gui      0.00%               1.77MiB / 3.814GiB    0.05%               234kB / 229kB       5.14MB / 4.1kB      2
36a212620e76        app_core     0.00%               0B / 0B               0.00%               0B / 0B             0B / 0B             0
8b1cb459328d        mongo               0.00%               0B / 0B               0.00%               0B / 0B             0B / 0B             0
dfe65081e256        watchtower          0.00%               5.105MiB / 3.814GiB   0.13%               22.4kB / 1.18kB     9.49MB / 0B         10

Dockerfile:

FROM arm32v7/node
ENV NODE_VERSION 11.15.0
WORKDIR ./

# install dependencies
RUN apt-get update;\
    apt-get install qemu qemu-user-static binfmt-support -y;\
    apt-get install nano -y;\
    apt-get install i2c-tools -y;

# nodejs packages
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./
RUN npm install --only=production

# copy app
COPY  src/ ./

# add logs folder
RUN mkdir -p /logs/

# wait for mongoDB launch
ADD wait /wait
RUN chmod +x /wait

EXPOSE 3000

STOPSIGNAL SIGTERM

CMD wait && npm run start

我有四个容器。通常,核心中断,我注意到mongocore有时有时会同时停止。 core依赖于mongo,但是gui依赖于core(并且不会崩溃),因此我不确定一个触发器是否触发另一个触发器。

core的docker日志仅指出: `以代码137退出

我在应用程序中添加了一个SIGTERM侦听器,该侦听器未向日志添加任何内容:

  process.on('SIGTERM', () => {
    console.log('Received SIGTERM shutting down');
    server.close( () => {
      process.exit(0);
    });
  });

问题:我该如何调试?我尝试过docker inspect,但不确定要寻找什么。

编辑:

我的应用程序中存在内存泄漏,但此刻上升非常缓慢。尽管如此,在退出容器之前,看起来内存仍然可以使用。

命令:

while true; do { printf "\n===docker====\n" && docker stats -a --no-stream && printf "\n----free----\n" && free && printf "\n----top----\n" && top -n 1 -b | grep Mem; } >> stats.txt; done

输出:

===docker====
    CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
41a88987cec4        app_gui      0.00%               1.715MiB / 3.814GiB   0.04%               634kB / 2.93MB      7.85MB / 0B         2
e7f90e0adaf2        app_core     74.80%              181.3MiB / 3.814GiB   4.64%               942kB / 479kB       26MB / 4.1kB        24
aebb3307cbfa        mongo               0.88%               29.9MiB / 3.814GiB    0.77%               330kB / 856kB       25.5MB / 36.2MB     21
7251a48141c5        watchtower          0.00%               4.34MiB / 3.814GiB    0.11%               19kB / 689B         631kB / 0B          9

----free----              total        used        free      shared  buff/cache   available
Mem:        3999784      704444     2733448        8864      561892     3161648
Swap:        102396           0      102396

----top----
    MiB Mem :   3906.0 total,   2668.6 free,    687.3 used,    550.1 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.   3088.1 avail Mem

===docker====
    CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
41a88987cec4        app_gui      0.00%               1.672MiB / 3.814GiB   0.04%               635kB / 2.93MB      7.85MB / 0B         2
e7f90e0adaf2        app_core     100.01%             84.27MiB / 3.814GiB   2.16%               942kB / 479kB       26MB / 4.1kB        15
aebb3307cbfa        mongo               1.69%               29.88MiB / 3.814GiB   0.76%               330kB / 856kB       25.5MB / 36.2MB     21
7251a48141c5        watchtower          0.00%               4.238MiB / 3.814GiB   0.11%               19kB / 689B         631kB / 0B          9

----free----              total        used        free      shared  buff/cache   available
Mem:        3999784      254364     3179764        8792      565656     3612488
Swap:        102396           0      102396

----top----
    MiB Mem :   3906.0 total,   3105.0 free,    248.1 used,    552.9 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.   3528.1 avail Mem

===docker====
    CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
41a88987cec4        app_gui      0.00%               1.672MiB / 3.814GiB   0.04%               635kB / 2.93MB      7.85MB / 0B         2
e7f90e0adaf2        app_core     0.00%               0B / 0B               0.00%               0B / 0B             0B / 0B             0
aebb3307cbfa        mongo               0.74%               29.83MiB / 3.814GiB   0.76%               330kB / 856kB       25.5MB / 36.2MB     15
7251a48141c5        watchtower          0.00%               4.238MiB / 3.814GiB   0.11%               19kB / 689B         631kB / 0B          9

----free----              total        used        free      shared  buff/cache   available
Mem:        3999784      252176     3180248        8792      567360     3614564
Swap:        102396           0      102396

----top----
    MiB Mem :   3906.0 total,   3105.1 free,    246.8 used,    554.1 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.   3529.3 avail Mem 

在容器退出之前,CPU使用率短于100%以上看起来很有趣。我知道发生泄漏时,Nodejs垃圾收集器会增加。

docker system df

TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              13                  4                   2.547GB             1.994GB (78%)
Containers          4                   3                   4.171kB             4.169kB (99%)
Local Volumes       6                   2                   0B                  0B
Build Cache         0                   0                   0B                  0B

编辑2)

还有另一种衡量标准,这次是CPU使用率很高,但在容器退出之前不超过100%:

===docker====
    CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
41a88987cec4        app_gui      0.00%               1.672MiB / 3.814GiB   0.04%               748kB / 2.93MB      7.85MB / 0B         2
e7f90e0adaf2        app_core     76.62%              183.9MiB / 3.814GiB   4.71%               503kB / 205kB       26.4MB / 4.1kB      24
aebb3307cbfa        mongo               0.70%               30.36MiB / 3.814GiB   0.78%               770kB / 1.4MB       25.5MB / 38.3MB     20
7251a48141c5        watchtower          0.00%               3.742MiB / 3.814GiB   0.10%               130kB / 689B        823kB / 0B          11

----free----
    total        used        free      shared  buff/cache   available
Mem:        3999784      678460     2750504       17064      570820     3177724
Swap:        102396           0      102396

----top----
    MiB Mem :   3906.0 total,   2699.5 free,    663.5 used,    543.1 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.   3104.1 avail Mem

===docker====
    CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
41a88987cec4        app_gui      0.00%               1.668MiB / 3.814GiB   0.04%               748kB / 2.93MB      7.85MB / 0B         2
e7f90e0adaf2        app_core     80.70%              2.621MiB / 3.814GiB   0.07%               505kB / 207kB       26.6MB / 8.19kB     0
aebb3307cbfa        mongo               1.16%               30.32MiB / 3.814GiB   0.78%               772kB / 1.4MB       25.6MB / 38.3MB     15
7251a48141c5        watchtower          0.00%               3.684MiB / 3.814GiB   0.09%               130kB / 689B        823kB / 0B          11

----free----
    total        used        free      shared  buff/cache   available
Mem:        3999784      233208     3160796       17000      605780     3625148
Swap:        102396           0      102396

----top----
    MiB Mem :   3906.0 total,   3084.8 free,    228.3 used,    592.9 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.   3539.8 avail Mem

===docker====
    CONTAINER ID        NAME                CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
41a88987cec4        app_gui      0.00%               1.668MiB / 3.814GiB   0.04%               748kB / 2.93MB      7.85MB / 0B         2
e7f90e0adaf2        app_core     0.00%               0B / 0B               0.00%               0B / 0B             0B / 0B             0
aebb3307cbfa        mongo               2.13%               30.32MiB / 3.814GiB   0.78%               773kB / 1.4MB       25.6MB / 38.3MB     15
7251a48141c5        watchtower          0.00%               3.684MiB / 3.814GiB   0.09%               130kB / 689B        823kB / 0B          11

----free----
    total        used        free      shared  buff/cache   available
Mem:        3999784      233492     3156400       16992      609892     3625140
Swap:        102396           0      102396

----top----
    MiB Mem :   3906.0 total,   3082.3 free,    228.1 used,    595.6 buff/cache
MiB Swap:    100.0 total,    100.0 free,      0.0 used.   3540.1 avail Mem 

应该剩下足够的内存,容器使用4.71%的内存,并且没有交换。

0 个答案:

没有答案