Question

我已经配置了一个流浪汉支持的kubernetes集群，但是我无法从节点上运行的pod中访问在master上运行的kube-apiserver。我试图通过api从pod中查找服务，但看起来api一直在断开连接。

使用pod中的curl我得到以下输出

root@itest-pod-2:/# curl -v \
--insecure -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
https://$KUBERNETES_SERVICE_HOST:443/api/v1/namespaces/default/services?labelSelector=name%3Dtest-server
* Hostname was NOT found in DNS cache
*   Trying 10.245.0.1...
* Connected to 10.245.0.1 (10.245.0.1) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to 10.245.0.1:443 
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to 10.245.0.1:443 
root@itest-pod-2:/#

但是，如果我通过简单地将所有节点组件安装到主服务器上来配置单个计算机环境，我就可以从一个pod中联系api。

root@itest-pod-3:/# curl -v --insecure \
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
https://$KUBERNETES_SERVICE_HOST:443/api/v1/namespaces/default/services?labelSelector=name%3Dtest-server
* Hostname was NOT found in DNS cache
*   Trying 10.245.0.1...
* Connected to 10.245.0.1 (10.245.0.1) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-SHA
* Server certificate:
*    subject: CN=10.0.2.15@1452869292
*    start date: 2016-01-15 14:48:12 GMT
*    expire date: 2017-01-14 14:48:12 GMT
*    issuer: CN=10.0.2.15@1452869292
*    SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /api/v1/namespaces/default/services?labelSelector=name%3Dtest-server HTTP/1.1
> User-Agent: curl/7.38.0
> Host: 10.245.0.1
> Accept: */*
> Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6ImRlZmF1bHQtdG9rZW4tdDY3cXUiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZGVmYXVsdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImIxNGI4YWE3LWJiOTgtMTFlNS1iNjhjLTA4MDAyN2FkY2NhZiIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmRlZmF1bHQifQ.HhPnit7Sfv-yUkMW6Cy9ZVbuiV2wt5PLaPSP-uZtaByOPagkb8d-8zBQE8Lx53lqxMFwBmjjxSWl-vKtSGa0ka6rEkq_xWtFJb8uDDlxz_R63R6IJBWB8YhwB7SzPVWgtHohj55D3pL8-r8NOQSQVXFAHaiGTlzmtwiE3CmJv3yBzBLALG0yvtW2YgwrO9jlxCGdFIOKae-5eduiOyZHUimxAgfBkbwSNfSzXYZTJNryfPiDBKZybh9c3Wd-pNsSZyw9gbBhbGFM7EiK9pWsdViQ__fZA2JbxX78YbajWE6CeL4FWLKFu4MuIlnmhLOvOXia_9WXz1B8XJ-MlzclZQ
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 15 Jan 2016 16:37:40 GMT
< Content-Length: 171
< 
{
  "kind": "ServiceList",
  "apiVersion": "v1",
  "metadata": {
    "selfLink": "/api/v1/namespaces/default/services",
    "resourceVersion": "1518"
  },
  "items": []
}
* Connection #0 to host 10.245.0.1 left intact

让我感到困惑的是，除了已经将节点组件安装到主服务器中之外，两种情况下的配置都是相同的，这让我觉得它不是ssl / https的错误配置，因为它是与kubernetes网络配置有关。

我已查看了apiserver的日志，但我看不到与这些丢弃的连接有关的任何内容。

非常感谢任何帮助。

Answer 1

我假设您将kubelet传递给worker / minion证书/密钥对。

api-server证书是否包含指向主服务器的可分辨名称/备用名称（openssl.conf上的subjectAltName在生成API服务器证书时应具有主服务器的IP）

这是我认为这个问题最常见的原因。

创建您的CA密钥和证书然后：

cat <<EOT >openssl.cnf
[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
[req_distinguished_name]
[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = kubernetes
DNS.2 = kubernetes.default
IP.1 = ${K8S_SERVICE_IP}
IP.2 = ${MASTER_HOST}
EOT

openssl genrsa -out apiserver-key.pem 2048
openssl req -new -key apiserver-key.pem -out apiserver.csr -subj "/CN=kube-apiserver" -config ./openssl.cnf
openssl x509 -req -in apiserver.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out apiserver.pem -days 365 -extensions v3_req -extfile ./openssl.cnf

Answer 2

问题是我们没有设置绑定地址 apiserver（我们设置了不安全的绑定地址，但没有设置--bind-address）我们认为这不会成为问题，因为默认情况下，apiserver会在所有接口上绑定。

当绑定到所有接口时，对/ api / v1 / endpoints的调用将返回apiserver安全端口的eth0 IP。在大多数情况下，这可能会很好但是因为我们在虚拟机上运行kubernetes vm eth0就是由virtualbox创建的NAT接口，可以通过它来访问 VBoxHeadless正在侦听的主机端口。

当传出流量离开pod时，它会触发一组iptables规则匹配集群服务ips并重定向到代理上的端口代理然后将请求转发给实际的机器群集。

在这种情况下，kube-proxy没有可用的 apiservice的外部可达ip，而无法访问virtualbox使用的eth0地址。

奇怪的是，似乎代理然后尝试联系 api在它不安全的端口上（它知道它的预定目的地）请求由于它创建的iptables规则）。自我们的要求在这种情况下是https，apiserver在第一个客户端之后丢弃它你好。

通常在卷曲中，这看起来像这样

root@app-master-0:/home/vagrant# curl -v --insecure \
https://10.235.1.2:8080/api/v1/namespaces/default/services?labelSelector=name%3Dtest-server
* Hostname was NOT found in DNS cache
*   Trying 10.235.1.2...
* Connected to 10.235.1.2 (10.235.1.2) port 8080 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
* Closing connection 0
curl: (35) error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown
protocol

但是当通过kube代理进行代理时，它看起来像这样

root@itest-pod-2:/# curl -v --insecure \
https://$KUBERNETES_SERVICE_HOST:443/api/v1/namespaces/default/services?labelSelector=name%3Dtest-server
* Hostname was NOT found in DNS cache
*   Trying 10.245.0.1...
* Connected to 10.245.0.1 (10.245.0.1) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* Unknown SSL protocol error in connection to 10.245.0.1:443
* Closing connection 0
curl: (35) Unknown SSL protocol error in connection to 10.245.0.1:443

通过将--bind-address=xxxx与外部可访问的eth1 ip添加到apiserver的args，我们能够解决此问题。

Kubernetes无法从节点

2 个答案: