Kubernetes Pod无法与出站通信

时间:2019-11-17 20:01:57

标签: kubernetes

我已经在运行CentOS 7的一组VM上安装了Kubernetes v1.13.10。当我部署Pod时,它们可以彼此连接,但不能连接到集群以外的任何设备。 CoreDNS pod的日志中有以下错误:

[ERROR] plugin/errors: 2 app.harness.io.xentaurs.com. A: unreachable backend: read udp 172.21.0.33:48105->10.20.10.52:53: i/o timeout
 [ERROR] plugin/errors: 2 app.harness.io.xentaurs.com. AAAA: unreachable backend: read udp 172.21.0.33:49098->10.20.10.51:53: i/o timeout
 [ERROR] plugin/errors: 2 app.harness.io.xentaurs.com. AAAA: unreachable backend: read udp 172.21.0.33:53113->10.20.10.51:53: i/o timeout
 [ERROR] plugin/errors: 2 app.harness.io.xentaurs.com. A: unreachable backend: read udp 172.21.0.33:39648->10.20.10.51:53: i/o timeout

IP 10.20.10.51和10.20.10.52是内部DNS服务器,可以从节点访问。我从DNS服务器进行了Wireshark捕获,我看到流量是从CoreDNS pod IP地址172.21.0.33传入的。 DNS服务器无法返回该IP,因为它无法在Kubernetes集群外部路由。

我的理解是,当Pod尝试进行出站通信时,应该实施iptables规则以将Pod IP定位到节点的地址(正确吗?)。以下是iptables中的POSTROUTING链:

[root@kube-aci-1 ~]# iptables -t nat -L POSTROUTING -v --line-number
Chain POSTROUTING (policy ACCEPT 23 packets, 2324 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1     1990  166K KUBE-POSTROUTING  all  --  any    any     anywhere             anywhere             /* kubernetes postrouting rules */
2        0     0 MASQUERADE  all  --  any    ens192.152  172.21.0.0/16        anywhere  

第1行是kube-proxy添加的,第2行是我手动添加的行,以尝试将来自Pod子网172.21.0.0/16的任何内容添加到节点接口ens192.152,但这没有用。

以下是kube-proxy日志:

[root@kube-aci-1 ~]# kubectl logs kube-proxy-llq22 -n kube-system
W1117 16:31:59.225870       1 proxier.go:498] Failed to load kernel module ip_vs with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1117 16:31:59.232006       1 proxier.go:498] Failed to load kernel module ip_vs_rr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1117 16:31:59.233727       1 proxier.go:498] Failed to load kernel module ip_vs_wrr with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1117 16:31:59.235700       1 proxier.go:498] Failed to load kernel module ip_vs_sh with modprobe. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules
W1117 16:31:59.255278       1 server_others.go:296] Flag proxy-mode="" unknown, assuming iptables proxy
I1117 16:31:59.289360       1 server_others.go:148] Using iptables Proxier.
I1117 16:31:59.296021       1 server_others.go:178] Tearing down inactive rules.
I1117 16:31:59.324352       1 server.go:484] Version: v1.13.10
I1117 16:31:59.335846       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I1117 16:31:59.336443       1 config.go:102] Starting endpoints config controller
I1117 16:31:59.336466       1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I1117 16:31:59.336493       1 config.go:202] Starting service config controller
I1117 16:31:59.336499       1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I1117 16:31:59.436617       1 controller_utils.go:1034] Caches are synced for service config controller
I1117 16:31:59.436739       1 controller_utils.go:1034] Caches are synced for endpoints config controller

我尝试刷新iptables nat表以及在所有节点上重新启动kube-proxy,但问题仍然存在。上面输出中的任何线索,或对进一步故障排除的想法?

kubectl get节点的输出:

[root@kube-aci-1 ~]# kubectl get nodes -o wide
NAME         STATUS   ROLES    AGE   VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION          CONTAINER-RUNTIME
kube-aci-1   Ready    master   85d   v1.13.10   10.10.52.217   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://1.13.1
kube-aci-2   Ready    <none>   85d   v1.13.10   10.10.52.218   <none>        CentOS Linux 7 (Core)   3.10.0-957.el7.x86_64   docker://1.13.1

1 个答案:

答案 0 :(得分:1)

事实证明,如果需要从Pod进行出站通信,则必须使用正在使用CNI的网络上可路由的子网。我使子网可以在外部网络上路由,并且Pod现在可以进行出站通信了。