k8s svc在其他工作程序节点中找不到Pod

时间:2019-07-03 02:35:50

标签: curl kubernetes

我有两个相同的pod在两个工作节点上运行,这些节点通过svc从外部进行服务。如下:

root@master1:~# kubectl get pod -o wide
NAME          READY   STATUS    RESTARTS   AGE   IP            NODE
kubia-nwjcc   1/1     Running   0          33m   10.244.1.27   worker1
kubia-zcpbb   1/1     Running   0          33m   10.244.2.11   worker2

root@master1:~# kubectl get svc
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1     <none>        443/TCP   26h
kubia        ClusterIP   10.98.41.49   <none>        80/TCP    34m

但是,当我尝试访问svc之一中的pod时,我只能在同一节点上获得pod的响应。当svc访问另一个节点上的pod时。当svc尝试访问其他节点上的pod时,它将返回command terminated with exit code 7。正确的输出和不良的输出似乎是随机生成的,如下所示

正确的输出

root@master1:~# k exec kubia-nwjcc -- curl http://10.98.41.49
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    23    0    23    0     0   8543      0 --:--:-- --:--:-- --:--:-- 11500
You've hit kubia-nwjcc

输出不良

root@master1:~# kubectl exec kubia-nwjcc -- curl http://10.98.41.49   
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
curl: (7) Failed to connect to 10.98.41.49 port 80: No route to host
command terminated with exit code 7

以下是我正在使用的软件版本:

  • ubuntu:v18.04
  • kubelet / kubeadm / kubectl:v1.15.0
  • docker:v18.09.5

以下是svc的描述:

root@master1:~# kubectl describe svc kubia
Name:              kubia
Namespace:         default
Labels:            <none>
Annotations:       <none>
Selector:          app=kubia
Type:              ClusterIP
IP:                10.98.41.49
Port:              <unset>  80/TCP
TargetPort:        8080/TCP
Endpoints:         10.244.1.27:8080,10.244.2.11:8080
Session Affinity:  None
Events:            <none>

以下是使用-v=9返回的结果:

root@master1:~# kubectl exec kubia-nwjcc -v=9 -- curl -s http://10.98.41.49
I0702 11:45:52.481239   23171 loader.go:359] Config loaded from file:  /root/.kube/config
I0702 11:45:52.501154   23171 round_trippers.go:419] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" 'https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc'
I0702 11:45:52.525926   23171 round_trippers.go:438] GET https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc 200 OK in 24 milliseconds
I0702 11:45:52.525980   23171 round_trippers.go:444] Response Headers:
I0702 11:45:52.525992   23171 round_trippers.go:447]     Content-Type: application/json
I0702 11:45:52.526003   23171 round_trippers.go:447]     Content-Length: 2374
I0702 11:45:52.526012   23171 round_trippers.go:447]     Date: Tue, 02 Jul 2019 11:45:52 GMT
I0702 11:45:52.526063   23171 request.go:947] Response Body: {"kind":"Pod","apiVersion":"v1","metadata":{"name":"kubia-nwjcc","generateName":"kubia-","namespace":"default","selfLink":"/api/v1/namespaces/default/pods/kubia-nwjcc","uid":"2fd67789-c48d-4459-8b03-ac562b4a3f5c","resourceVersion":"188689","creationTimestamp":"2019-07-02T10:51:34Z","labels":{"app":"kubia"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"kubia","uid":"f3a4c457-dee4-4aec-ad73-1f0ca41628aa","controller":true,"blockOwnerDeletion":true}]},"spec":{"volumes":[{"name":"default-token-6pgh8","secret":{"secretName":"default-token-6pgh8","defaultMode":420}}],"containers":[{"name":"kubia","image":"luksa/kubia","ports":[{"containerPort":8080,"protocol":"TCP"}],"resources":{},"volumeMounts":[{"name":"default-token-6pgh8","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always"}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"worker1","securityContext":{},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:15Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:20Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-03T01:35:20Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2019-07-02T10:51:34Z"}],"hostIP":"192.168.56.21","podIP":"10.244.1.27","startTime":"2019-07-03T01:35:15Z","containerStatuses":[{"name":"kubia","state":{"running":{"startedAt":"2019-07-03T01:35:19Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"luksa/kubia:latest","imageID":"docker-pullable://luksa/kubia@sha256:3f28e304dc0f63dc30f273a4202096f0fa0d08510bd2ee7e1032ce600616de24","containerID":"docker://27da556930baf857e5af92b13934dcb1b2b2f001ecab5e7b952b2bda5aa27f0b"}],"qosClass":"BestEffort"}}
I0702 11:45:52.543108   23171 round_trippers.go:419] curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.15.0 (linux/amd64) kubernetes/e8462b5" 'https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc/exec?command=curl&command=-s&command=http%3A%2F%2F10.98.41.49&container=kubia&stderr=true&stdout=true'
I0702 11:45:52.591166   23171 round_trippers.go:438] POST https://192.168.56.11:6443/api/v1/namespaces/default/pods/kubia-nwjcc/exec?command=curl&command=-s&command=http%3A%2F%2F10.98.41.49&container=kubia&stderr=true&stdout=true 101 Switching Protocols in 47 milliseconds
I0702 11:45:52.591208   23171 round_trippers.go:444] Response Headers:
I0702 11:45:52.591217   23171 round_trippers.go:447]     Connection: Upgrade
I0702 11:45:52.591221   23171 round_trippers.go:447]     Upgrade: SPDY/3.1
I0702 11:45:52.591225   23171 round_trippers.go:447]     X-Stream-Protocol-Version: v4.channel.k8s.io
I0702 11:45:52.591229   23171 round_trippers.go:447]     Date: Wed, 03 Jul 2019 02:29:33 GMT
F0702 11:45:53.783725   23171 helpers.go:114] command terminated with exit code 7

kube-system吊舱和提供服务的两个吊舱的状态均为Running,如下所示:

root@master1:~/k8s-yaml# kubectl get --all-namespaces pod -o wide
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE     IP              NODE      NOMINATED NODE   READINESS GATES
default       kubia-6pjz9                       1/1     Running   0          5m35s   10.244.2.12     worker2   <none>           <none>
default       kubia-nwjcc                       1/1     Running   0          16h     10.244.1.27     worker1   <none>           <none>
kube-system   coredns-bccdc95cf-792px           1/1     Running   4          5d19h   10.244.0.11     master1   <none>           <none>
kube-system   coredns-bccdc95cf-bc76j           1/1     Running   4          5d19h   10.244.0.10     master1   <none>           <none>
kube-system   etcd-master1                      1/1     Running   8          5d19h   192.168.56.11   master1   <none>           <none>
kube-system   kube-apiserver-master1            1/1     Running   7          5d19h   192.168.56.11   master1   <none>           <none>
kube-system   kube-controller-manager-master1   1/1     Running   7          5d18h   192.168.56.11   master1   <none>           <none>
kube-system   kube-flannel-ds-amd64-9trbq       1/1     Running   3          5d18h   192.168.56.21   worker1   <none>           <none>
kube-system   kube-flannel-ds-amd64-btt74       1/1     Running   5          5d18h   192.168.56.11   master1   <none>           <none>
kube-system   kube-flannel-ds-amd64-swzzd       1/1     Running   0          3d20h   192.168.56.22   worker2   <none>           <none>
kube-system   kube-proxy-27zfk                  1/1     Running   7          5d19h   192.168.56.11   master1   <none>           <none>
kube-system   kube-proxy-79kt9                  1/1     Running   0          3d20h   192.168.56.22   worker2   <none>           <none>
kube-system   kube-proxy-lx4gk                  1/1     Running   3          5d18h   192.168.56.21   worker1   <none>           <none>
kube-system   kube-scheduler-master1            1/1     Running   7          5d19h   192.168.56.11   master1   <none>           <none>

我的预期回报是:使用curl访问svc时,svc可以随机访问不同节点上的pod,如下所示:

root@master1:~# kubectl get po
NAME          READY   STATUS    RESTARTS   AGE
kubia-nwjcc   1/1     Running   0          50m
kubia-zcpbb   1/1     Running   0          50m
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-zcpbb
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-zcpbb
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc

现在,五个测试的返回结果如下:

root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
command terminated with exit code 7
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
You've hit kubia-nwjcc
root@master1:~# k exec kubia-nwjcc -- curl -s http://10.98.41.49
command terminated with exit code 7

1 个答案:

答案 0 :(得分:1)

此问题已解决。 flannel的官方文档提到在--iface类型的虚拟机中运行时,需要使用vagrant来指定要使用的网卡。您可以使用命令kubectl edit daemonset kube-flannel-ds-amd64 -n kube-system编辑fllannel配置文件。然后对所有法兰绒荚使用kubectl delete pod -n kube-system <pod-name>。 K8会重建它们。

您可以在Kubernetes with Flannelflannel - troubleshooting中找到详细的答案。