COS的/ var / lib / docker卷的inode用完了

时间：2017-07-03 11:26:01

标签： docker google-cloud-platform google-container-os

我正在努力让我们在GCP上运行一些服务。

我目前看到的一个问题是，我很快启动的虚拟机似乎耗尽了/ var / lib / docker文件系统的inode。我原本以为这是在容器优化的操作系统中调整的事情之一吗？

wouter@nbwm-cron ~ $ df -hi Filesystem Inodes IUsed IFree IUse% Mounted on /dev/root 78K 13K 65K 17% / devtmpfs 463K 204 463K 1% /dev tmpfs 464K 1 464K 1% /dev/shm tmpfs 464K 500 463K 1% /run tmpfs 464K 13 464K 1% /sys/fs/cgroup tmpfs 464K 9 464K 1% /mnt/disks tmpfs 464K 16K 448K 4% /tmp /dev/sda8 4.0K 11 4.0K 1% /usr/share/oem /dev/sda1 1013K 998K 15K 99% /var tmpfs 464K 45 464K 1% /var/lib/cloud overlayfs 464K 39 464K 1% /etc

wouter@nbwm-cron ~ $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE <name>/stackdriver-agent latest 0c4b075e7550 3 days ago 1.423 GB <none> <none> 96d027d3feea 4 days ago 905.2 MB gcr.io/<project>/nbwm-ops/docker-php5 latest 5d2c59c7dd7a 2 weeks ago 1.788 GB

nbwm-cron wouter # tune2fs -l /dev/sda1 tune2fs 1.43.3 (04-Sep-2016) Filesystem volume name: STATE Last mounted on: /var Filesystem UUID: ca44779b-ffd5-405a-bd3e-528071b45f73 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Remount read-only Filesystem OS type: Linux Inode count: 1036320 Block count: 4158971 Reserved block count: 0 Free blocks: 4062454 Free inodes: 1030756 First block: 0 Block size: 4096 Fragment size: 4096 Group descriptor size: 64 Reserved GDT blocks: 747 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8160 Inode blocks per group: 510 Flex block group size: 16 Filesystem created: Thu Jun 15 22:39:33 2017 Last mount time: Wed Jun 28 13:51:31 2017 Last write time: Wed Jun 28 13:51:31 2017 Mount count: 5 Maximum mount count: -1 Last checked: Thu Nov 19 19:00:00 2009 Check interval: 0 (<none>) Lifetime writes: 67 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 66aa0e7f-57da-41d0-86f7-d93270e53030 Journal backup: inode blocks

如何调整文件系统以使更多inode可用？

2 个答案:

答案 0 :(得分：1)

这是docker中overlay存储驱动程序的已知问题，由overlay2驱动程序解决。 新的cos-61 releases使用docker 17.03和overlay2存储驱动程序。你能试试看看问题是否会再次发生？

谢谢！

答案 1 :(得分：1)

我在GKE 57.9202.64.0（{1}}上的1.11.2（码头65.10323.85.0）和17.03.2（码头1.8.12-gke.1）上的所有COS版本都见过同样的问题{GEL {{1} }。旧版本肯定也会受到影响。

这些都使用overlay驱动程序：

pdecat@gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ docker info 2>&1 | grep "Storage Driver"
Storage Driver: overlay

pdecat@gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ grep "\(CHROMEOS_RELEASE_VERSION\|CHROMEOS_RELEASE_CHROME_MILESTONE\)" /etc/lsb-release
CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_VERSION=10323.85.0

overlay2驱动程序仅用于具有相同COS版本的GKE 1.9+群集（新增或升级）：

pdecat@gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ docker info 2>&1 | grep "Storage Driver"
Storage Driver: overlay2

pdecat@gke-cluster-test-pdecat-default-pool-e8945081-xhj6 ~ $ grep "\(CHROMEOS_RELEASE_VERSION\|CHROMEOS_RELEASE_CHROME_MILESTONE\)" /etc/lsb-release
CHROMEOS_RELEASE_CHROME_MILESTONE=65
CHROMEOS_RELEASE_VERSION=10323.85.0

当overlay驱动程序出现可用空间/ inode问题时，我使用spotify的docker-gc解决它：

# docker run --rm --userns host -v /var/run/docker.sock:/var/run/docker.sock -v /etc:/etc spotify/docker-gc

在：

# df -hi /var/lib/docker/
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda1        6.0M  5.0M  1.1M   83% /var

# df -h /var/lib/docker/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        95G   84G   11G  89% /var

# du --inodes -s /var/lib/docker/*
180     /var/lib/docker/containers
4093    /var/lib/docker/image
4       /var/lib/docker/network
4906733 /var/lib/docker/overlay
1       /var/lib/docker/tmp
1       /var/lib/docker/trust
25      /var/lib/docker/volumes

后：

# df -hi /var/lib/docker/
Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda1        6.0M  327K  5.7M    6% /var/lib/docker

# df -h /var/lib/docker/
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        95G  6.6G   88G   7% /var/lib/docker

# du --inodes -s /var/lib/docker/*
218     /var/lib/docker/containers
1792    /var/lib/docker/image
4       /var/lib/docker/network
279002  /var/lib/docker/overlay
1       /var/lib/docker/tmp
1       /var/lib/docker/trust
25      /var/lib/docker/volumes

注意：使用通常的docker rmi $(docker images --filter "dangling=true" -q --no-trunc)和docker rm $(docker ps -qa --no-trunc --filter "status=exited")无法帮助恢复/var/lib/docker/overlay中的资源。

相关问题
最新问题