在docker中运行kubernetes时,秘密默认令牌挂载泄漏

时间:2016-04-22 13:49:54

标签: kubernetes

从本指南开始kubernetes之后:http://kubernetes.io/docs/getting-started-guides/docker/我的节点上有很多未使用的挂载点。这似乎取决于正在运行的pod数量。刚才我必须卸载超过2600个挂载点。当这些构建时,它会导致findmnt占用大量资源。挂载条目如下所示:

tmpfs on /var/lib/kubelet/pods/599d6157-081e-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha type tmpfs (rw)

有人知道为什么这些没有自动卸载吗?从教程中可以预见到你必须清理其中的一些(在关闭群集部分下看),但这似乎过分了。几天前我不得不清理22,000左右,因为我有一个mongo集群,redis运行了一段时间。

---更新--- 在清除我未使用的坐骑系统并等待几分钟之后,findmnt会生成如下条目:

├─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha             tmpfs                           tmpfs               rw,relatime
│ └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha           tmpfs                           tmpfs               rw,relatime
│   └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha         tmpfs                           tmpfs               rw,relatime
│     └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha       tmpfs                           tmpfs               rw,relatime
│       └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha     tmpfs                           tmpfs               rw,relatime
│         └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha   tmpfs                           tmpfs               rw,relatime
│           └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs                           tmpfs               rw,relatime
├─/var/lib/docker/containers/c84ad9b0f2ec580bedef394aa46bb147ed6c4f1e9454cd3729459d9127c0986e/shm                         shm                             tmpfs               rw,nosuid,nodev,noe
├─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia             tmpfs                           tmpfs               rw,relatime
│ └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia           tmpfs                           tmpfs               rw,relatime
│   └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia         tmpfs                           tmpfs               rw,relatime
│     └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia       tmpfs                           tmpfs               rw,relatime
│       └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia     tmpfs                           tmpfs               rw,relatime
│         └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia   tmpfs                           tmpfs               rw,relatime
│           └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia tmpfs                           tmpfs               rw,relatime
├─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha             tmpfs                           tmpfs               rw,relatime
│ └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha           tmpfs                           tmpfs               rw,relatime
│   └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha         tmpfs                           tmpfs               rw,relatime
│     └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha       tmpfs                           tmpfs               rw,relatime
│       └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha     tmpfs                           tmpfs               rw,relatime
│         └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha   tmpfs                           tmpfs               rw,relatime
│           └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs                           tmpfs               rw,relatime
├─/var/lib/docker/containers/5392d49f5140274ddcfbe757cf6a07336aa60975f3ea122d865a3b80f5540c1f/shm  

- 更新#2 - 这就是我启动kubelet的方式

ARCH=amd64
DNS_IP=10.0.0.10
K8S_VERSION=$(curl -sS https://storage.googleapis.com/kubernetes-release/release/stable.txt)


docker run \
    --volume=/:/rootfs:ro \
    --volume=/sys:/sys:ro \
    --volume=/var/lib/docker/:/var/lib/docker:rw \
    --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
    --volume=/var/run:/var/run:rw \
    --net=host \
    --pid=host \
    --privileged=true \
    --name=kubelet \
    -d \
    gcr.io/google_containers/hyperkube-${ARCH}:${K8S_VERSION} \
    /hyperkube kubelet \
        --containerized \
        --hostname-override="127.0.0.1" \
        --address="0.0.0.0" \
        --api-servers=http://localhost:8080 \
        --config=/etc/kubernetes/manifests \
        --cluster-dns=$DNS_IP \
        --cluster-domain=cluster.local \
        --allow-privileged=true --v=2

在查看其他一些建议时(感谢Thibault Deheurles),我尝试删除--containerized和--volume = /:/ rootfs:ro,但这导致k8s根本无法启动。

- 更新#3 - 我尝试添加mount标志,共享到我的/ var / lib / kubelet volume命令,它现在看起来像这样:

  

- 体积=在/ var / lib中/ kubelet /:在/ var / lib中/ kubelet:RW,共享

这并没有什么不同。

但是,我注意到在追踪我的kubelet docker容器的日志时,每次我获得一个新的挂载时都会出现此消息......

2016-04-26T20:30:52.447842722Z I0426 20:30:52.447559   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1

GC记录失败也会出现在日志中...以下是一些条目

2016-04-26T20:38:11.436858475Z E0426 20:38:11.436757   21740 kubelet.go:956] Image garbage collection failed: non-existent label "docker-images"
2016-04-26T20:38:12.448049454Z I0426 20:38:12.447852   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1
2016-04-26T20:38:52.448175137Z I0426 20:38:52.447949   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/d95a6048198f747c5fcb74ee23f1f25c/volumes/kubernetes.io~empty-dir/data: exit status 1
2016-04-26T20:39:14.447892769Z I0426 20:39:14.447649   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1
2016-04-26T20:39:28.441137221Z I0426 20:39:28.440920   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1
2016-04-26T20:40:20.441118739Z I0426 20:40:20.441018   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/d95a6048198f747c5fcb74ee23f1f25c/volumes/kubernetes.io~empty-dir/data: exit status 1
2016-04-26T20:40:22.447832573Z I0426 20:40:22.447590   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1
2016-04-26T20:40:53.447612605Z I0426 20:40:53.447534   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1
2016-04-26T20:41:27.449053007Z I0426 20:41:27.448820   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/d95a6048198f747c5fcb74ee23f1f25c/volumes/kubernetes.io~empty-dir/data: exit status 1
2016-04-26T20:41:30.440974280Z I0426 20:41:30.440889   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1
2016-04-26T20:41:58.441001603Z I0426 20:41:58.440906   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1

- 更新#4 - @PaulMorie询问了mount / findmnt版本

$ which findmnt
/bin/findmnt

$ uname -a
Linux andromeda 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:    14.04
Codename:   trusty

$ dpkg -L mount | grep findmn
/usr/share/man/man8/findmnt.8.gz
/bin/findmnt

$ dpkg -l mount
ii  mount                                2.20.1-5.1ubuntu20.7    amd64                   Tools for mounting and manipulating filesystems

- 更新#5 - @tsaarni问我做了什么来解决这个问题...这是我的黑客

[eric@andromeda [feature/k8s-packaging-openvpn]util]$ cat clean-mounts.sh 
#!/bin/bash
counter=0
for  m in $( mount| grep secret | awk '{print $3}' ); do 
    sudo umount $m
    counter=$[counter + 1]
done
echo "cleaned $counter mounts"


[eric@andromeda [feature/k8s-packaging-openvpn]util]$ cat clean-mounts-watcher.sh 
#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)"
while : ; do $DIR/clean-mounts.sh ; sleep 60; done

1 个答案:

答案 0 :(得分:0)

docker教程暂时已过时。您需要添加一些关于如何运行不同服务的元素。

我的最后一个工作版本是this one