我有一个kubernetes集群(使用法兰绒):
kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.5", GitCommit:"51dd616cdd25d6ee22c83a858773b607328a18ec", GitTreeState:"clean", BuildDate:"2019-01-16T18:14:49Z", GoVersion:"go1.10.7", Compiler:"gc", Platform:"linux/amd64"}
kubectl get cs
NAME STATUS MESSAGE ERROR
scheduler Healthy ok
controller-manager Healthy ok
etcd-0 Healthy {"health": "true"}
一切似乎都正常运行
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-576cbf47c7-q7ncm 1/1 Running 1 30m
coredns-576cbf47c7-tclp8 1/1 Running 1 30m
etcd-kube1 1/1 Running 1 30m
kube-apiserver-kube1 1/1 Running 1 30m
kube-controller-manager-kube1 1/1 Running 1 30m
kube-flannel-ds-amd64-6vlkx 1/1 Running 1 30m
kube-flannel-ds-amd64-7twk8 1/1 Running 1 30m
kube-flannel-ds-amd64-rqzr7 1/1 Running 1 30m
kube-proxy-krfzk 1/1 Running 1 30m
kube-proxy-vrssw 1/1 Running 1 30m
kube-proxy-xlrgz 1/1 Running 1 30m
kube-scheduler-kube1 1/1 Running 1 30m
现在,我已经部署了2个Pod(没有服务)。 2个NGinx吊舱。我还创建了一个busybox窗格。当我从busybox容器内部卷曲到同一节点上的nginx容器时,它会起作用:
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE
busybox 1/1 Running 2 30m 10.244.2.5 kube2 <none>
nginx-d55b94fd-l7swz 1/1 Running 1 30m 10.244.2.4 kube2 <none>
nginx-d55b94fd-zg7sj 1/1 Running 1 30m 10.244.1.6 kube3 <none>
卷曲:
kubectl exec busybox -- curl 10.244.2.4
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 612 100 612 0 0 357k 0 --:--:-- --:--:-- --:--:-- 597k
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
但是当我将Pod卷曲在另一个节点上时:
kubectl exec busybox -- curl 10.244.1.6
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- 10.244.1.6 port 80: No route to host
我该如何调试?有什么事吗(所有防火墙均已关闭/禁用)
其他信息:
$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
$ sysctl net.bridge.bridge-nf-call-iptables
net.bridge.bridge-nf-call-iptables = 1
信息:
kubectl exec -it busybox ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
3: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue
link/ether 0a:58:0a:f4:02:05 brd ff:ff:ff:ff:ff:ff
inet 10.244.2.5/24 scope global eth0
valid_lft forever preferred_lft forever
Iptables:
vagrant@kube1:~$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-EXTERNAL-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL all -- anywhere anywhere
Chain FORWARD (policy DROP)
target prot opt source destination
KUBE-FORWARD all -- anywhere anywhere /* kubernetes forwarding rules */
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- kube1/16 anywhere
ACCEPT all -- anywhere kube1/16
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL all -- anywhere anywhere
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
Chain KUBE-EXTERNAL-SERVICES (1 references)
target prot opt source destination
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- anywhere anywhere /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
Chain KUBE-FORWARD (1 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT all -- kube1/16 anywhere /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT all -- anywhere kube1/16 /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED
Chain KUBE-SERVICES (1 references)
target prot opt source destination
答案 0 :(得分:1)
我该如何调试?有什么事吗(所有防火墙均已关闭/禁用)
如果您在节点上禁用了iptables,则可能是问题。覆盖图(绒布)设置iptables,以允许Pod到Pod的流量。您可以使用以下方法检查K8s节点:
iptables-save | grep 10.244.2.4
iptables-save | grep 10.244.2.5
iptables-save | grep 10.244.2.6
您应该看到端口80
的一些规则:
-A KUBE-SEP-XXXXXXXXXXXXXXXX -s 10.244.2.4/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-XXXXXXXXXXXXXXXX -p tcp -m tcp -j DNAT --to-destination 10.244.2.4:80
答案 1 :(得分:0)
最有可能是iptables问题。
运行以下命令并检查
答案 2 :(得分:0)
确保在所有节点中都有这些规则
iptables -t filter -L KUBE-FORWARD --line-numbers -n
Chain KUBE-FORWARD (1 references)
num target prot opt source destination
1 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ mark match 0x4000/0x4000
2 ACCEPT all -- 10.42.0.0/16 0.0.0.0/0 /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
3 ACCEPT all -- 0.0.0.0/0 10.42.0.0/16 /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED