从本指南开始kubernetes之后:http://kubernetes.io/docs/getting-started-guides/docker/我的节点上有很多未使用的挂载点。这似乎取决于正在运行的pod数量。刚才我必须卸载超过2600个挂载点。当这些构建时,它会导致findmnt占用大量资源。挂载条目如下所示:
tmpfs on /var/lib/kubelet/pods/599d6157-081e-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha type tmpfs (rw)
有人知道为什么这些没有自动卸载吗?从教程中可以预见到你必须清理其中的一些(在关闭群集部分下看),但这似乎过分了。几天前我不得不清理22,000左右,因为我有一个mongo集群,redis运行了一段时间。
---更新--- 在清除我未使用的坐骑系统并等待几分钟之后,findmnt会生成如下条目:
├─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
├─/var/lib/docker/containers/c84ad9b0f2ec580bedef394aa46bb147ed6c4f1e9454cd3729459d9127c0986e/shm shm tmpfs rw,nosuid,nodev,noe
├─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia tmpfs tmpfs rw,relatime
├─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
│ └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs tmpfs rw,relatime
├─/var/lib/docker/containers/5392d49f5140274ddcfbe757cf6a07336aa60975f3ea122d865a3b80f5540c1f/shm
- 更新#2 - 这就是我启动kubelet的方式
ARCH=amd64
DNS_IP=10.0.0.10
K8S_VERSION=$(curl -sS https://storage.googleapis.com/kubernetes-release/release/stable.txt)
docker run \
--volume=/:/rootfs:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:rw \
--volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
--volume=/var/run:/var/run:rw \
--net=host \
--pid=host \
--privileged=true \
--name=kubelet \
-d \
gcr.io/google_containers/hyperkube-${ARCH}:${K8S_VERSION} \
/hyperkube kubelet \
--containerized \
--hostname-override="127.0.0.1" \
--address="0.0.0.0" \
--api-servers=http://localhost:8080 \
--config=/etc/kubernetes/manifests \
--cluster-dns=$DNS_IP \
--cluster-domain=cluster.local \
--allow-privileged=true --v=2
在查看其他一些建议时(感谢Thibault Deheurles),我尝试删除--containerized和--volume = /:/ rootfs:ro,但这导致k8s根本无法启动。
- 更新#3 - 我尝试添加mount标志,共享到我的/ var / lib / kubelet volume命令,它现在看起来像这样:
- 体积=在/ var / lib中/ kubelet /:在/ var / lib中/ kubelet:RW,共享
这并没有什么不同。
但是,我注意到在追踪我的kubelet docker容器的日志时,每次我获得一个新的挂载时都会出现此消息......
2016-04-26T20:30:52.447842722Z I0426 20:30:52.447559 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1
GC记录失败也会出现在日志中...以下是一些条目
2016-04-26T20:38:11.436858475Z E0426 20:38:11.436757 21740 kubelet.go:956] Image garbage collection failed: non-existent label "docker-images"
2016-04-26T20:38:12.448049454Z I0426 20:38:12.447852 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1
2016-04-26T20:38:52.448175137Z I0426 20:38:52.447949 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/d95a6048198f747c5fcb74ee23f1f25c/volumes/kubernetes.io~empty-dir/data: exit status 1
2016-04-26T20:39:14.447892769Z I0426 20:39:14.447649 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1
2016-04-26T20:39:28.441137221Z I0426 20:39:28.440920 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1
2016-04-26T20:40:20.441118739Z I0426 20:40:20.441018 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/d95a6048198f747c5fcb74ee23f1f25c/volumes/kubernetes.io~empty-dir/data: exit status 1
2016-04-26T20:40:22.447832573Z I0426 20:40:22.447590 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1
2016-04-26T20:40:53.447612605Z I0426 20:40:53.447534 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1
2016-04-26T20:41:27.449053007Z I0426 20:41:27.448820 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/d95a6048198f747c5fcb74ee23f1f25c/volumes/kubernetes.io~empty-dir/data: exit status 1
2016-04-26T20:41:30.440974280Z I0426 20:41:30.440889 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1
2016-04-26T20:41:58.441001603Z I0426 20:41:58.440906 21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1
- 更新#4 - @PaulMorie询问了mount / findmnt版本
$ which findmnt
/bin/findmnt
$ uname -a
Linux andromeda 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
$ dpkg -L mount | grep findmn
/usr/share/man/man8/findmnt.8.gz
/bin/findmnt
$ dpkg -l mount
ii mount 2.20.1-5.1ubuntu20.7 amd64 Tools for mounting and manipulating filesystems
- 更新#5 - @tsaarni问我做了什么来解决这个问题...这是我的黑客
[eric@andromeda [feature/k8s-packaging-openvpn]util]$ cat clean-mounts.sh
#!/bin/bash
counter=0
for m in $( mount| grep secret | awk '{print $3}' ); do
sudo umount $m
counter=$[counter + 1]
done
echo "cleaned $counter mounts"
[eric@andromeda [feature/k8s-packaging-openvpn]util]$ cat clean-mounts-watcher.sh
#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)"
while : ; do $DIR/clean-mounts.sh ; sleep 60; done