我有带有11个节点环境的k8,其中3个是主副本和2个主副本 我的k8在内部部署环境中提供了多种服务,我的主主节点停止暴露端口,而另外2个主节点(副本1,副本2)正在暴露我的Pod的端口
环境是:
例如,k8s仪表板暴露在所有节点上的端口30465中,但是只有副本1和2正在获得连接。
我们看到我的主要主人正在监听端口30465,但它没有将其重定向。
root@master-main:~# lsof -i:30465
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
kube-prox 32627 root 25u IPv6 7465821 0t0 TCP *:30465 (LISTEN)
所有我的大师都检查了ufw规则,它们是相同的
root@k8s-master-main:~# iptables -t nat -nL | grep dashboard
KUBE-MARK-MASQ tcp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kubernetes-dashboard: */ tcp dpt:30465
KUBE-SVC-XGLOHA7QRQ3V22RZ tcp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kubernetes-dashboard: */ tcp dpt:30465
KUBE-MARK-MASQ all -- 10.32.0.3 0.0.0.0/0 /* kube-system/kubernetes-dashboard: */
DNAT tcp -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kubernetes-dashboard: */ tcp to:10.32.0.3:8443
KUBE-MARK-MASQ tcp -- !10.32.0.0/12 10.110.60.225 /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:443
KUBE-SVC-XGLOHA7QRQ3V22RZ tcp -- 0.0.0.0/0 10.110.60.225 /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:443
KUBE-SEP-LPUGT7E25KUQ5PUI all -- 0.0.0.0/0 0.0.0.0/0 /* kube-system/kubernetes-dashboard: */
当我为特定端口运行tcpdup时,我发现没有流量进入。
root@k8s-master-main:~# tcpdump -ni any port 30465
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
13:01:13.002312 IP 192.168.10.139.65533 > 192.168.132.133.30465: Flags [S], seq 3169086234, win 64260, options [mss 1428,nop,wscale 8,nop,nop,sackOK], length 0
13:01:13.252722 IP 192.168.10.139.65535 > 192.168.132.133.30465: Flags [S], seq 1338864689, win 64260, options [mss 1428,nop,wscale 8,nop,nop,sackOK], length 0
13:01:16.000630 IP 192.168.10.139.65533 > 192.168.132.133.30465: Flags [S], seq 3169086234, win 64260, options [mss 1428,nop,wscale 8,nop,nop,sackOK], length 0
13:01:16.253443 IP 192.168.10.139.65535 > 192.168.132.133.30465: Flags [S], seq 1338864689, win 64260, options [mss 1428,nop,wscale 8,nop,nop,sackOK], length 0
我们可以看到相同的端口,当我在其上运行tcpdump时,我们看到流量进入了内部
root@k8s-master-replica1:~# tcpdump -ni any port 30465
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
13:06:15.891608 IP 192.168.10.139.49475 > 192.168.132.135.30465: Flags [S], seq 3874981346, win 64260, options [mss 1428,nop,wscale 8,nop,nop,sackOK], length 0
13:06:15.892994 IP 192.168.132.135.30465 > 192.168.10.139.49475: Flags [S.], seq 4062915301, ack 3874981347, win 26720, options [mss 1336,nop,nop,sackOK,nop,wscale 7], length 0
13:06:15.893576 IP 192.168.10.139.49475 > 192.168.132.135.30465: Flags [.], ack 1, win 260, length 0
13:06:15.895480 IP 192.168.10.139.49475 > 192.168.132.135.30465: Flags [P.], seq 1:518, ack 1, win 260, length 517
13:06:15.895658 IP 192.168.132.135.30465 > 192.168.10.139.49475: Flags [.], ack 518, win 218, length 0
13:06:15.895788 IP 192.168.132.135.30465 > 192.168.10.139.49475: Flags [P.], seq 1:147, ack 518, win 218, length 146
13:06:15.896432 IP 192.168.10.139.49475 > 192.168.132.135.30465: Flags [P.], seq 518:569, ack 147, win 260, length 51
13:06:15.896865 IP 192.168.132.135.30465 > 192.168.10.139.49475: Flags [P.], seq 147:203, ack 569, win 218, length 56
13:06:15.933856 IP 192.168.10.139.49475 > 192.168.132.135.30465: Flags [P.], seq 569:746, ack 203, win 260, length 177
13:06:15.933885 IP 192.168.10.139.49475 > 192.168.132.135.30465: Flags [P.], seq 746:1022, ack 203, win 260, length 276
我检查了我的 docker 是否能够在主主节点上使用,因此我将nginx图像拉出并暴露在外部,并且可以正常工作。
root@k8s-master-main:~# tcpdump -ni any port 8080
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
13:35:19.394587 IP 192.168.10..139.52032 > 192.168.132.133.8080: Flags [F.], seq 3885521030, ack 1729975292, win 252, length 0
13:35:19.394755 IP 192.168.132.133.8080 > 192.168.10..139.52032: Flags [F.], seq 1, ack 1, win 245, length 0
13:35:19.395317 IP 192.168.10..139.52032 > 192.168.132.133.8080: Flags [.], ack 2, win 252, length 0
13:35:19.428629 IP 192.168.10..139.52071 > 192.168.132.133.8080: Flags [S], seq 3014852904, win 64260, options [mss 1428,nop,wscale 8,nop,nop,sackOK], length 0
13:35:19.428724 IP 192.168.132.133.8080 > 192.168.10..139.52071: Flags [S.], seq 1168464434, ack 3014852905, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:35:19.430191 IP 192.168.10..139.52071 > 192.168.132.133.8080: Flags [.], ack 1, win 256, length 0
13:35:19.431712 IP 192.168.10..139.52071 > 192.168.132.133.8080: Flags [P.], seq 1:380, ack 1, win 256, length 379: HTTP: GET / HTTP/1.1
13:35:19.431751 IP 192.168.132.133.8080 > 192.168.10..139.52071: Flags [.], ack 380, win 237, length 0
13:35:19.431905 IP 192.168.132.133.8080 > 192.168.10..139.52071: Flags [P.], seq 1:239, ack 380, win 237, length 238: HTTP: HTTP/1.1 200 OK
13:35:19.431983 IP 192.168.132.133.8080 > 192.168.10..139.52071: Flags [P.], seq 239:851, ack 380, win 237, length 612: HTTP
13:35:19.433158 IP 192.168.10..139.52071 > 192.168.132.133.8080: Flags [.], ack 851, win 253, length 0
13:35:19.453609 IP 192.168.10..139.52071 > 192.168.132.133.8080: Flags [P.], seq 380:740, ack 851, win 253, length 360: HTTP: GET /favicon.ico HTTP/1.1
13:35:19.453927 IP 192.168.132.133.8080 > 192.168.10..139.52071: Flags [P.], seq 851:1159, ack 740, win 245, length 308: HTTP: HTTP/1.1 404 Not Found
13:35:19.498549 IP 192.168.10..139.52071 > 192.168.132.133.8080: Flags [.], ack 1159, win 252, length 0
谢谢