Kubelet-无法为coredns“ CreatePodSandbox”;无法设置网桥地址:无法将IP地址添加到“ cni0”:权限被拒绝

时间:2020-06-16 12:03:56

标签: kubernetes kubelet podman cni cri-o

编辑1

针对这些评论,我提供了更多信息。

$ kubectl get pods --namespace kube-system
NAME                                                  READY   STATUS              RESTARTS   AGE
coredns-66bff467f8-lkwfn                              0/1     ContainerCreating   0          7m8s
coredns-66bff467f8-pcn6b                              0/1     ContainerCreating   0          7m8s
etcd-masternode                                       1/1     Running             0          7m16s
kube-apiserver-masternode                             1/1     Running             0          7m16s
kube-controller-manager-masternode                    1/1     Running             0          7m16s
kube-proxy-7zrjn                                      1/1     Running             0          7m8s
kube-scheduler-masternode                             1/1     Running             0          7m16s

更多systemd日志

...
Jun 16 16:18:59 masternode kubelet[6842]: E0616 16:18:59.313433    6842 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-66bff467f8-pcn6b_kube-system_d5fe7a46-c32d-4fa3-b1b3-fe5a28983e08_0(cc72c59e22145274e47ca417c274af99591d0008baf2bf13364538b7debb57d3): failed to set bridge addr: could not add IP address to "cni0": permission denied
Jun 16 16:18:59 masternode kubelet[6842]: E0616 16:18:59.313512    6842 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "coredns-66bff467f8-pcn6b_kube-system(d5fe7a46-c32d-4fa3-b1b3-fe5a28983e08)" failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-66bff467f8-pcn6b_kube-system_d5fe7a46-c32d-4fa3-b1b3-fe5a28983e08_0(cc72c59e22145274e47ca417c274af99591d0008baf2bf13364538b7debb57d3): failed to set bridge addr: could not add IP address to "cni0": permission denied
Jun 16 16:18:59 masternode kubelet[6842]: E0616 16:18:59.313532    6842 kuberuntime_manager.go:727] createPodSandbox for pod "coredns-66bff467f8-pcn6b_kube-system(d5fe7a46-c32d-4fa3-b1b3-fe5a28983e08)" failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-66bff467f8-pcn6b_kube-system_d5fe7a46-c32d-4fa3-b1b3-fe5a28983e08_0(cc72c59e22145274e47ca417c274af99591d0008baf2bf13364538b7debb57d3): failed to set bridge addr: could not add IP address to "cni0": permission denied
Jun 16 16:18:59 masternode kubelet[6842]: E0616 16:18:59.313603    6842 pod_workers.go:191] Error syncing pod d5fe7a46-c32d-4fa3-b1b3-fe5a28983e08 ("coredns-66bff467f8-pcn6b_kube-system(d5fe7a46-c32d-4fa3-b1b3-fe5a28983e08)"), skipping: failed to "CreatePodSandbox" for "coredns-66bff467f8-pcn6b_kube-system(d5fe7a46-c32d-4fa3-b1b3-fe5a28983e08)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-66bff467f8-pcn6b_kube-system(d5fe7a46-c32d-4fa3-b1b3-fe5a28983e08)\" failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-66bff467f8-pcn6b_kube-system_d5fe7a46-c32d-4fa3-b1b3-fe5a28983e08_0(cc72c59e22145274e47ca417c274af99591d0008baf2bf13364538b7debb57d3): failed to set bridge addr: could not add IP address to \"cni0\": permission denied"
Jun 16 16:19:09 masternode kubelet[6842]: E0616 16:19:09.256408    6842 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-66bff467f8-lkwfn_kube-system_f0187bfd-89a2-474c-b843-b00875183c77_0(1aba005509e85f3ea7da3fc48ab789ae3a10ba0ffefc152d1c4edf65693befe2): failed to set bridge addr: could not add IP address to "cni0": permission denied
Jun 16 16:19:09 masternode kubelet[6842]: E0616 16:19:09.256498    6842 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "coredns-66bff467f8-lkwfn_kube-system(f0187bfd-89a2-474c-b843-b00875183c77)" failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-66bff467f8-lkwfn_kube-system_f0187bfd-89a2-474c-b843-b00875183c77_0(1aba005509e85f3ea7da3fc48ab789ae3a10ba0ffefc152d1c4edf65693befe2): failed to set bridge addr: could not add IP address to "cni0": permission denied
Jun 16 16:19:09 masternode kubelet[6842]: E0616 16:19:09.256525    6842 kuberuntime_manager.go:727] createPodSandbox for pod "coredns-66bff467f8-lkwfn_kube-system(f0187bfd-89a2-474c-b843-b00875183c77)" failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-66bff467f8-lkwfn_kube-system_f0187bfd-89a2-474c-b843-b00875183c77_0(1aba005509e85f3ea7da3fc48ab789ae3a10ba0ffefc152d1c4edf65693befe2): failed to set bridge addr: could not add IP address to "cni0": permission denied
Jun 16 16:19:09 masternode kubelet[6842]: E0616 16:19:09.256634    6842 pod_workers.go:191] Error syncing pod f0187bfd-89a2-474c-b843-b00875183c77 ("coredns-66bff467f8-lkwfn_kube-system(f0187bfd-89a2-474c-b843-b00875183c77)"), skipping: failed to "CreatePodSandbox" for "coredns-66bff467f8-lkwfn_kube-system(f0187bfd-89a2-474c-b843-b00875183c77)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-66bff467f8-lkwfn_kube-system(f0187bfd-89a2-474c-b843-b00875183c77)\" failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_coredns-66bff467f8-lkwfn_kube-system_f0187bfd-89a2-474c-b843-b00875183c77_0(1aba005509e85f3ea7da3fc48ab789ae3a10ba0ffefc152d1c4edf65693befe2): failed to set bridge addr: could not add IP address to \"cni0\": permission denied"
... (repeats over and over again)

我已经用CRI-0 1.18成功安装了Kubernetes 1.18,并使用kubeadm init --pod-network-cidr = 192.168.0.0 / 16设置了一个集群。但是,“ coredns”节点被卡在“ ContainerCreating”上。我遵循了Kubernetes官方安装说明。

我尝试过的

我尝试安装Calico,但没有解决问题。我也尝试过手动将cni0接口更改为UP,但这也没有用。问题显然出在流量桥接的地方,但是我遵循了Kubernetes教程并启用了它。

在研究问题时,我偶然找到了有前途的解决方案和教程,但没有一个解决问题。 (Rancher GitHub IssueCRI-O GitHub PageProjectcalicoKubernetes tutorial

防火墙cmd

$ sudo firewall-cmd --state
running
$ sudo firewall-cmd --version
0.7.0

系统日志

Image of the log 因为粘贴整个日志会很丑陋。

uname -r

4.18.0-147.8.1.el8_1.x86_64 (Centos 8)

CRI-O

crio --version
crio version
Version:       1.18.1
GitCommit:     5cbf694c34f8d1af19eb873e39057663a4830635
GitTreeState:  clean
BuildDate:     2020-05-25T19:01:44Z
GoVersion:     go1.13.4
Compiler:      gc
Platform:      linux/amd64
Linkmode:      dynamic

runc

$ runc --version
runc version spec: 1.0.1-dev

Kubernetes

1.18

Podman版本

1.6.4

iptables / nft

我在iptables兼容性层上使用了nft。

$ iptables --version
iptables v1.8.2 (nf_tables)

主机提供商:

Contabo VPS

sysctl

$ sysctl net.bridge
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0
$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

已禁用selinux

$ cat /etc/sysconfig/selinux

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of these three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

ip地址列表

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether REDACTED brd ff:ff:ff:ff:ff:ff
    inet REDACTED scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
3: cni0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether c6:00:41:85:da:ad brd ff:ff:ff:ff:ff:ff
    inet 10.85.0.1/16 brd 10.85.255.255 scope global noprefixroute cni0
       valid_lft forever preferred_lft forever
7: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1440 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
    inet 192.168.249.128/32 brd 192.168.249.128 scope global tunl0
       valid_lft forever preferred_lft forever

2 个答案:

答案 0 :(得分:1)

安提阿的圣手榴弹!我终于解决了!仅仅花了我大约一个十亿一千万年的时间和一个不安的夜晚。甜蜜的胜利!好吧...嗯进入解决方案。

我终于理解@Arghya Sadhu和@Piotr Malec的评论,他们是对的。我没有正确配置CNI插件。我使用Flannel作为网络提供商,他们需要10.244.0.0/16子网。在我的在/etc/cni/net.d/中找到的crio-bridge.conf中,默认子网不同(10.85.0.0/16之类)。我认为在kubeadm init命令上指定CIDR就足够了,但是我错了。您需要在crio-bridge.conf和podman.conflist(或目录中的类似文件)中设置正确的CIDR。我还认为用CRI-O安装的那些文件都配置有合理的默认值,老实说,我并不完全了解它们的用途。

还发生了一些奇怪的事情:根据Flannel的说法,CRI-O的子网应该是/ 16,但是当我使用journalctl -u kubelet检查日志时,它提到了/ 24子网。

failed to set bridge addr: \"cni0\" already has an IP address different from 10.244.0.1/24"

因此,我不得不将crio.conf中的子网更改为/ 24,并且它可以工作。我可能也必须在podman.conflist中更改子网,但是我不确定。

无论如何,感谢Arghya和Piotr的帮助!

答案 1 :(得分:0)

要使用 Calico 网络插件和 cri-o 容器运行时设置集群,我必须:

添加到/etc/crio/crio.conf

[crio.network]
network_dir = "/etc/cni/net.d/"
plugin_dirs = [
    "/opt/cni/bin/",
    "/usr/libexec/cni/",
]

--cgroup-driver=systemd中添加/var/lib/kubelet/kubeadm-flags.env

KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd --container-runtime=remote --container-runtime-endpoint=/var/run/crio/crio.sock --pod-infra-container-image=k8s.gcr.io/pause:3.5"

重启kubeletcrio

systemctl daemon-reload && systemctl restart kubelet crio

初始化集群

kubeadm init --pod-network-cidr='10.85.0.0/16'

安装 calico 网络插件

kubectl create -f https://docs.projectcalico.org/manifests/calico.yaml