“ kubectl exec”导致“错误:无法升级连接:pod不存在”

时间:2018-07-03 12:40:14

标签: kubernetes kubeadm

root@master2:/home/osboxes# kubectl get pods --all-namespaces -o wide
NAMESPACE     NAME                                READY     STATUS    RESTARTS   AGE       IP             NODE
default       hello-kubernetes-55857678b4-4xbgd   1/1       Running   1          18h       10.244.1.138   node
default       hello-kubernetes-55857678b4-gtvn4   1/1       Running   1          18h       10.244.1.139   node
default       hello-kubernetes-55857678b4-wttht   1/1       Running   1          18h       10.244.1.140   node
kube-system   coredns-78fcdf6894-s4l8n            1/1       Running   1          18h       10.244.0.14    master2
kube-system   coredns-78fcdf6894-tfjps            1/1       Running   1          18h       10.244.0.15    master2
kube-system   etcd-master2                        1/1       Running   1          18h       10.0.2.15      master2
kube-system   kube-apiserver-master2              1/1       Running   1          18h       10.0.2.15      master2
kube-system   kube-controller-manager-master2     1/1       Running   1          18h       10.0.2.15      master2
kube-system   kube-flannel-ds-4br99               1/1       Running   1          18h       10.0.2.15      node
kube-system   kube-flannel-ds-6c2x9               1/1       Running   1          18h       10.0.2.15      master2
kube-system   kube-proxy-mf9fg                    1/1       Running   1          18h       10.0.2.15      node
kube-system   kube-proxy-xldph                    1/1       Running   1          18h       10.0.2.15      master2
kube-system   kube-scheduler-master2              1/1       Running   1          18h       10.0.2.15      master2
root@master2:/home/osboxes# kubectl exec -it hello-kubernetes-55857678b4-4xbgd sh
error: unable to upgrade connection: pod does not exist

此错误表示什么?我可以从节点docker exec ...进入容器。

我已经使用kubeadm设置了这个集群。

详细:

kubectl -v=10 exec -it hello-kubernetes-55857678b4-4xbgd sh
I0703 08:44:01.250752   10307 loader.go:359] Config loaded from file /etc/kubernetes/admin.conf
I0703 08:44:01.252809   10307 loader.go:359] Config loaded from file /etc/kubernetes/admin.conf
I0703 08:44:01.254167   10307 loader.go:359] Config loaded from file /etc/kubernetes/admin.conf
I0703 08:44:01.255808   10307 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.11.0 (linux/amd64) kubernetes/91e7b4f" 'https://192.168.0.33:6443/api/v1/namespaces/default/pods/hello-kubernetes-55857678b4-4xbgd'
I0703 08:44:01.272882   10307 round_trippers.go:405] GET https://192.168.0.33:6443/api/v1/namespaces/default/pods/hello-kubernetes-55857678b4-4xbgd 200 OK in 16 milliseconds
I0703 08:44:01.273262   10307 round_trippers.go:411] Response Headers:
I0703 08:44:01.273485   10307 round_trippers.go:414]     Date: Tue, 03 Jul 2018 12:44:01 GMT
I0703 08:44:01.273692   10307 round_trippers.go:414]     Content-Type: application/json
I0703 08:44:01.273967   10307 round_trippers.go:414]     Content-Length: 2725
I0703 08:44:01.275168   10307 request.go:897] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"hello-kubernetes-55857678b4-4xbgd","generateName":"hello-kubernetes-55857678b4-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/hello-kubernetes-55857678b4-4xbgd","uid":"002c6e23-7e23-11e8-b38f-0800273a59cb","resourceVersion":"5725","creationTimestamp":"2018-07-02T18:09:02Z","labels":{"app":"hello-kubernetes","pod-template-hash":"1141323460"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"hello-kubernetes-55857678b4","uid":"001893c6-7e23-11e8-b38f-0800273a59cb","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-n9t9d","secret":{"secretName":"default-token-n9t9d","defaultMode":420}}],"containers":[{"name":"hello-kubernetes","image":"paulbouwer/hello-kubernetes:1.4","ports":[{"containerPort":8080,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"default-token-n9t9d","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"node","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}]},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-07-02T18:09:02Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-07-03T12:32:26Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":null},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2018-07-02T18:09:02Z"}],"hostIP":"10.0.2.15","podIP":"10.244.1.138","startTime":"2018-07-02T18:09:02Z","containerStatuses":[{"name":"hello-kubernetes","state":{"running":{"startedAt":"2018-07-03T12:32:26Z"}},"lastState":{"terminated":{"exitCode":255,"reason":"Error","startedAt":"2018-07-02T18:09:21Z","finishedAt":"2018-07-03T10:10:58Z","containerID":"docker://e82d0338a51aef35869b755b8020704367859855f043d80897e48f4e9c7da869"}},"ready":true,"restartCount":1,"image":"paulbouwer/hello-kubernetes:1.4","imageID":"docker-pullable://paulbouwer/hello-kubernetes@sha256:a9fc93acfbc734827a72107bf7f759745a66ea61758863c094c36e5f4f4b810b","containerID":"docker://4a7e472b35b776700e61605826655950501d114ce182dc178d79d0f50775281d"}],"qosClass":"BestEffort"}}
I0703 08:44:01.290627   10307 round_trippers.go:386] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.11.0 (linux/amd64) kubernetes/91e7b4f" 'https://192.168.0.33:6443/api/v1/namespaces/default/pods/hello-kubernetes-55857678b4-4xbgd/exec?command=sh&container=hello-kubernetes&container=hello-kubernetes&stdin=true&stdout=true&tty=true'
                                                                                                               I0703 08:44:01.317914   10307 round_trippers.go:405] POST https://192.168.0.33:6443/api/v1/namespaces/default/pods/hello-kubernetes-55857678b4-4xbgd/exec?command=sh&container=hello-kubernetes&container=hello-kubernetes&stdin=true&stdout=true&tty=true 404 Not Found in 26 milliseconds
                                                                                                                                                                                         I0703 08:44:01.317938   10307 round_trippers.go:411] Response Headers:
                                              I0703 08:44:01.317944   10307 round_trippers.go:414]     Date: Tue, 03 Jul 2018 12:44:01 GMT
                                                                                                                                          I0703 08:44:01.317948   10307 round_trippers.go:414]     Content-Length: 18
    I0703 08:44:01.317951   10307 round_trippers.go:414]     Content-Type: text/plain; charset=utf-8
                                                                                                    F0703 08:44:01.318118   10307 helpers.go:119] error: unable to upgrade connection: pod does not exist

2 个答案:

答案 0 :(得分:1)

似乎使用了错误的网络接口。

我必须在节点和主节点上的KUBELET_EXTRA_ARGS=--node-ip=ABCXYZ中手动设置/etc/default/kubernetes(用适当的IP地址替换abcxyz)。

您可以通过以下方法检查它们的IP地址是否正确:

kubectl get nodes -o wide 

哪个输出:

NAME      STATUS    ROLES     AGE       VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION      CONTAINER-RUNTIME
master2   Ready     master    19h       v1.11.0   192.168.0.33   <none>        Ubuntu 17.10   4.13.0-46-generic   docker://1.13.1
node      Ready     <none>    18h       v1.11.0   192.168.0.34   <none>        Ubuntu 17.10   4.13.0-16-generic   docker://1.13.1

答案 1 :(得分:0)

TL;DR

通过键入“kubectl get nodes -o wide”来测试集群上的任何两个节点是否使用相同的 IP 地址。如果有,请设置新的修复 IP,或者在启用 DHCP 的情况下,获取新 IP 执行

sudo dhclient -r yourNetworkInterface

sudo dhclient yourNetworkInterface

问题

将新的工作节点加入正在运行的 Kubernetes 集群后,无法再在新节点加入集群后创建的容器上成功执行“kubectl exec”,而是弹出错误消息:"error: unable to upgrade connection: pod does not exist"

enter image description here

kubectl get pods -A 表明有问题的 pod 正在新加入的节点上运行。

enter image description here

原因

kubectl get nodes -A 显示主节点和工作节点使用相同的 IP 地址。

enter image description here

解决方案

为工作节点分配一个新的 IP 地址,如果启用了 DHCP,当然可以使用 dhclient 或使用 netplan 修复新的 IP

enter image description here