我正在尝试安装kubernetes。 检查在kubernetes上执行的容器时出现问题。 我将服务类型设置为Nodeport，但我无法从运行容器的节点以外的节点进行访问。 想要从其他计算机访问它，请告诉我它的不同之处。 我尝试过externalIPs和LoadBarancer，但这是不可能的。
环境
网络
机器
kubectl描述了svc nginx-cluster
Name: nginx-cluster
Namespace: default
Labels: app=nginx-demo
Annotations: <none>
Selector: app=nginx-demo
Type: ClusterIP
IP: 172.16.236.159
Port: <unset> 8090/TCP
TargetPort: 80/TCP
Endpoints: 192.168.24.2:80
Session Affinity: None
Events: <none>
kubectl描述了svc nginx-service
Name: nginx-service
Namespace: default
Labels: app=nginx-demo
Annotations: <none>
Selector: app=nginx-demo
Type: NodePort
IP: 172.16.199.69
Port: <unset> 8090/TCP
TargetPort: 80/TCP
NodePort: <unset> 31659/TCP
Endpoints: 192.168.24.2:80
Session Affinity: None
External Traffic Policy: Cluster
Events: <none>
运行容器工作节点（10.1.1.27）
卷曲10.1.1.27:31659
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...
工作节点（10.1.1.26）
卷曲10.1.1.27:31659
curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.
其他机器（10.1.1.XX）
卷曲10.1.1.27:31659
curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.
kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
echoserver-848b75d85-9fx7r 1/1 Running 3 6d 192.168.70.2 k8swrksv01
nginx-demo-85cc49574c-wv2b9 1/1 Running 3 6d 192.168.2.2 k8swrksv02
kubectl获得svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
clusterip ClusterIP 172.16.39.77 <none> 80/TCP 6d run=echoserver
kubernetes ClusterIP 172.16.0.1 <none> 443/TCP 10d <none>
nginx-cluster ClusterIP 172.16.236.159 <none> 8090/TCP 6d app=nginx-demo
nginx-service NodePort 172.16.199.69 <none> 8090:31659/TCP 6d app=nginx-demo
nodeport NodePort 172.16.38.40 <none> 80:31317/TCP 6d run=echoserver
netstat -ntlp
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 1963/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 2202/kube-proxy
tcp 0 0 127.0.0.1:4243 0.0.0.0:* LISTEN 1758/dockerd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 996/sshd
tcp6 0 0 :::4194 :::* LISTEN 1963/kubelet
tcp6 0 0 :::10250 :::* LISTEN 1963/kubelet
tcp6 0 0 :::31659 :::* LISTEN 2202/kube-proxy
tcp6 0 0 :::10255 :::* LISTEN 1963/kubelet
tcp6 0 0 :::10256 :::* LISTEN 2202/kube-proxy
tcp6 0 0 :::31317 :::* LISTEN 2202/kube-proxy
tcp6 0 0 :::22 :::* LISTEN 996/sshd
的iptables保存
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-AZ4EGFEAU4RTSLJO - [0:0]
:KUBE-SEP-C7HQKKO26GIFOZZM - [0:0]
:KUBE-SEP-EWKNS2YCPXGJCXDC - [0:0]
:KUBE-SEP-LQVPUPFGW6BWATIP - [0:0]
:KUBE-SEP-OMMOFZ27GPKZ4OPA - [0:0]
:KUBE-SEP-UD3HOGDD5NDLNY74 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-CQNAS6RSUGJF2C2D - [0:0]
:KUBE-SVC-GKN7Y2BSGW4NJTYL - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-XP7QDA4CRQ2QA33W - [0:0]
:KUBE-SVC-Z5P6OMNAEVLAQUTS - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 192.168.2.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.0.0/16 -d 192.168.0.0/16 -j RETURN
-A POSTROUTING -s 192.168.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.2.0/24 -j RETURN
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -s 192.168.70.2/32 -m comment --comment "default/clusterip:" -j KUBE-MARK-MASQ
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -p tcp -m comment --comment "default/clusterip:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-C7HQKKO26GIFOZZM -s 192.168.70.2/32 -m comment --comment "default/nodeport:" -j KUBE-MARK-MASQ
-A KUBE-SEP-C7HQKKO26GIFOZZM -p tcp -m comment --comment "default/nodeport:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-EWKNS2YCPXGJCXDC -s 10.1.1.25/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-EWKNS2YCPXGJCXDC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.25:6443
-A KUBE-SEP-LQVPUPFGW6BWATIP -s 192.168.2.2/32 -m comment --comment "default/nginx-service:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m comment --comment "default/nginx-service:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -s 10.1.1.24/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.24:6443
-A KUBE-SEP-UD3HOGDD5NDLNY74 -s 192.168.2.2/32 -m comment --comment "default/nginx-cluster:" -j KUBE-MARK-MASQ
-A KUBE-SEP-UD3HOGDD5NDLNY74 -p tcp -m comment --comment "default/nginx-cluster:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SERVICES -d 172.16.236.159/32 -p tcp -m comment --comment "default/nginx-cluster: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-Z5P6OMNAEVLAQUTS
-A KUBE-SERVICES -d 172.16.199.69/32 -p tcp -m comment --comment "default/nginx-service: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-SERVICES -d 172.16.38.40/32 -p tcp -m comment --comment "default/nodeport: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-SERVICES -d 172.16.39.77/32 -p tcp -m comment --comment "default/clusterip: cluster IP" -m tcp --dport 80 -j KUBE-SVC-CQNAS6RSUGJF2C2D
-A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-CQNAS6RSUGJF2C2D -m comment --comment "default/clusterip:" -j KUBE-SEP-AZ4EGFEAU4RTSLJO
-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m comment --comment "default/nginx-service:" -j KUBE-SEP-LQVPUPFGW6BWATIP
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-XP7QDA4CRQ2QA33W -m comment --comment "default/nodeport:" -j KUBE-SEP-C7HQKKO26GIFOZZM
-A KUBE-SVC-Z5P6OMNAEVLAQUTS -m comment --comment "default/nginx-cluster:" -j KUBE-SEP-UD3HOGDD5NDLNY74
COMMIT
*filter
:INPUT ACCEPT [40:14606]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [42:6275]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:DOCKER-USER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
答案 0 :(得分：3)
它是由FORWARD链上的默认DROP引起的（而这又是由docker引起的）。
如果您向节点添加iptables -A FORWARD -j ACCEPT规则，则可以再次看到它。
k8s问题在这里：https://github.com/kubernetes/kubernetes/issues/39823但实际修复程序在https://github.com/kubernetes/kubernetes/pull/52569（预计在1.9中）。
答案 1 :(得分：0)
其他信息。 有些东西应该阻止端口，但这是未知的......
pod运行节点
nmap 10.1.1.27 -p31000-32000
Not shown: 999 closed ports
PORT STATE SERVICE
31317/tcp open unknown
31659/tcp open unknown
其他节点
nmap 10.1.1.27 -p31000-32000
Not shown: 999 closed ports
PORT STATE SERVICE
31317/tcp filtered unknown
31659/tcp filtered unknown
答案 2 :(得分：0)
Farcaller是对的。我们在运行firewalld的同时也达到了同样的效果。
在我们升级到k8s 1.9之前，我们添加了以下firewalld规则。该规则类似于k8s 1.9
中由kube-proxy创建的规则#!/bin/bash
# follows https://github.com/kubernetes/kubernetes/pull/52569 introduced in k8s 1.9
# required to support nodeport services routing from all nodes in the cluster when the firewall is turned on.
# KUBE-MARK-MASQ corresponds to kube-proxy --iptables-masquerade-bit=14, which is the default.
KUBE_MARK_MASQ="0x4000/0x4000"
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 3 -m mark --mark "$KUBE_MARK_MASQ" -j ACCEPT
答案 3 :(得分：-2)
您可以先检查节点端口端口是否打开，
使用
netstat -ntlp
检查，如果是，可能是iptable或路由的一些问题，
如果不是，请检查防火墙或其他问题
祝你好运