我有一个由 1 个经理、3 个工作人员组成的小型集群设置来试验一些事情。它运行 swarm 编排，能够从任何堆栈 yaml 启动跨集群的服务，并通过入口网络为 web 应用程序提供服务。我没有对 docker-ce 的默认 yum 安装进行任何更改。 Vanilla 安装，无需更改任何节点的配置。
但是，存在通过其他覆盖网络进行服务间通信的问题。 我创建了一个带有 --attachable 标志的 docker 覆盖网络 testnet 并在 node-1 上将一个 nginx（名为：nginx1）容器附加到它，在 manager-1 上将一个 netshoot（名为：netshoot1）容器附加到它.
然后我可以从 netshoot1 ping nginx1，反之亦然。我可以在两个节点上通过 tcpdump 观察到这些数据包交换。
# tcpdump -vvnn -i any src 10.1.72.70 and dst 10.1.72.71 and port 4789
00:20:39.302561 IP (tos 0x0, ttl 64, id 49791, offset 0, flags [none], proto UDP (17), length 134)
10.1.72.70.53237 > 10.1.72.71.4789: [udp sum ok] VXLAN, flags [I] (0x08), vni 4101
IP (tos 0x0, ttl 64, id 20598, offset 0, flags [DF], proto ICMP (1), length 84)
10.0.5.18 > 10.0.5.24: ICMP echo request, id 21429, seq 1, length 64
这里可以看到 netshoot1 (10.0.5.18) ping nginx1 (10.0.5.24) - echo 成功。
但是，如果我 # curl -v nginx1:80，整个过程就会超时。
使用 tcpdump，我可以看到数据包离开 manager-1 节点，但它们永远不会到达节点 1。
00:22:22.809057 IP (tos 0x0, ttl 64, id 42866, offset 0, flags [none], proto UDP (17), length 110)
10.1.72.70.53764 > 10.1.72.71.4789: [bad udp cksum 0x5b97 -> 0x697d!] VXLAN, flags [I] (0x08), vni 4101
IP (tos 0x0, ttl 64, id 43409, offset 0, flags [DF], proto TCP (6), length 60)
10.0.5.18.53668 > 10.0.5.24.80: Flags [S], cksum 0x1e58 (incorrect -> 0x2c3e), seq 1616566654, win 28200, options [mss 1410,sack OK,TS val 913132903 ecr 0,nop,wscale 7], length 0
这些是通过 vmware 在内部数据中心上运行的虚拟机。网络团队表示网络防火墙不应阻止或检查它们，因为 ip 位于同一子网上。
这是 docker 配置的问题吗？ iptables？
操作系统：RHEL 8
Docker CE：20.10.2
容器：1.4.3
manager-1 上的 IPTABLES
Chain INPUT (policy DROP 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 9819K 2542M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
2 8 317 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 255
3 473 33064 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
4 0 0 DROP all -- * * 127.0.0.0/8 0.0.0.0/0
5 116 6192 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
6 351K 21M ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 source IP range 10.1.72.71-10.1.72.73 state NEW multiport dports 2377,7946
7 435 58400 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 source IP range 10.1.72.71-10.1.72.73 state NEW multiport dports 7946,4789
8 17142 1747K REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain FORWARD (policy DROP 8 packets, 384 bytes)
num pkts bytes target prot opt in out source destination
1 14081 36M DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
2 14081 36M DOCKER-INGRESS all -- * * 0.0.0.0/0 0.0.0.0/0
3 267K 995M DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
4 39782 121M ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
5 1598 95684 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
6 41470 717M ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
7 0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
8 90279 23M ACCEPT all -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
9 5 300 DOCKER all -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0
10 94041 134M ACCEPT all -- docker_gwbridge !docker_gwbridge 0.0.0.0/0 0.0.0.0/0
11 0 0 DROP all -- docker_gwbridge docker_gwbridge 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 11M packets, 2365M bytes)
num pkts bytes target prot opt in out source destination
Chain DOCKER (2 references)
num pkts bytes target prot opt in out source destination
1 1598 95684 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:5000
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num pkts bytes target prot opt in out source destination
1 41470 717M DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
2 93853 133M DOCKER-ISOLATION-STAGE-2 all -- docker_gwbridge !docker_gwbridge 0.0.0.0/0 0.0.0.0/0
3 267K 995M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
num pkts bytes target prot opt in out source destination
1 1033K 1699M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-INGRESS (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8502
2 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:8502
3 267K 995M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
2 0 0 DROP all -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0
3 135K 851M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
节点 1 上的 IPTABLES
Chain INPUT (policy DROP 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 6211K 3343M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
2 7 233 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmptype 255
3 471 32891 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
4 0 0 DROP all -- * * 127.0.0.0/8 0.0.0.0/0
5 84 4504 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22 /* ssh from anywhere */
6 26940 1616K ACCEPT tcp -- * * 10.1.72.70 0.0.0.0/0 state NEW multiport dports 7946 /* docker swarm cluster comm- manager,node2,3 */
7 31624 1897K ACCEPT tcp -- * * 10.1.72.72 0.0.0.0/0 state NEW multiport dports 7946 /* docker swarm cluster comm- manager,node2,3 */
8 30583 1835K ACCEPT tcp -- * * 10.1.72.73 0.0.0.0/0 state NEW multiport dports 7946 /* docker swarm cluster comm- manager,node2,3 */
9 432 58828 ACCEPT udp -- * * 10.1.72.70 0.0.0.0/0 state NEW multiport dports 7946,4789 /* docker swarm cluster comm and overlay netw- manager,node2,3 */
10 10 1523 ACCEPT udp -- * * 10.1.72.72 0.0.0.0/0 state NEW multiport dports 7946,4789 /* docker swarm cluster comm and overlay netw- manager,node2,3 */
11 7 1159 ACCEPT udp -- * * 10.1.72.73 0.0.0.0/0 state NEW multiport dports 7946,4789 /* docker swarm cluster comm and overlay netw- manager,node2,3 */
12 17172 1749K REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain FORWARD (policy DROP 19921 packets, 1648K bytes)
num pkts bytes target prot opt in out source destination
1 23299 22M DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0
2 23299 22M DOCKER-INGRESS all -- * * 0.0.0.0/0 0.0.0.0/0
3 787K 1473M DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0
4 0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
5 0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
6 0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
7 0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
8 386K 220M ACCEPT all -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
9 0 0 DOCKER all -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0
10 402K 1254M ACCEPT all -- docker_gwbridge !docker_gwbridge 0.0.0.0/0 0.0.0.0/0
11 0 0 DROP all -- docker_gwbridge docker_gwbridge 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 8193K packets, 2659M bytes)
num pkts bytes target prot opt in out source destination
Chain DOCKER-INGRESS (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8502
2 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:8502
3 787K 1473M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
num pkts bytes target prot opt in out source destination
1 792K 1474M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER (2 references)
num pkts bytes target prot opt in out source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 DOCKER-ISOLATION-STAGE-2 all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
2 402K 1254M DOCKER-ISOLATION-STAGE-2 all -- docker_gwbridge !docker_gwbridge 0.0.0.0/0 0.0.0.0/0
3 787K 1473M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 DROP all -- * docker0 0.0.0.0/0 0.0.0.0/0
2 0 0 DROP all -- * docker_gwbridge 0.0.0.0/0 0.0.0.0/0
3 402K 1254M RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
答案 0 :(得分：2)
问题确实是出站数据包上的错误校验和。由于校验和错误，vmware 网络接口丢弃了数据包。
解决方案是禁用校验和卸载。使用 ethtool：
# ethtool -K <interface> tx off