Question

我已将kubernetes升级到版本1.1.7并从我的一个pod中获得此错误，该pod经常调用k8s ApiServer来检查其他每个pod的活跃状态。

Error #01: Get http://[api-server]:8080/api/v1/namespaces/production/pods?labelSelector=app%3Dworkflow-worker-mandrill-hook-handler: dial tcp [api-server]:8080: connect: cannot assign requested address

请求以每秒约80个请求的速率发送。虽然有这个错误，我仍然能够从我的本地调用该API。重新启动pod解决了问题，但第二天，它再次发生了。似乎apiserver阻止该pod以避免DOS？

我正在使用docker版本Docker version 1.7.1, build 2c2c52b-dirty和CoreOS v773.0.0

Linux ***** 4.1.5-coreos #2 SMP Thu Aug 13 09:18:45 UTC 2015 x86_64 Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz GenuineIntel GNU/Linux

Kubernetes api服务器错误日志：

I0306 07:32:13.087599       1 logs.go:40] http: TLS handshake error from ***:60033: EOF
I0306 07:32:14.596398       1 logs.go:40] http: TLS handshake error from ***:57257: EOF
I0306 07:32:15.126962       1 logs.go:40] http: TLS handshake error from ***:60035: EOF
I0306 07:32:15.136445       1 logs.go:40] http: TLS handshake error from ***:60054: EOF
I0306 07:32:15.210656       1 logs.go:40] http: TLS handshake error from ***:45384: EOF
I0306 07:32:15.215155       1 logs.go:40] http: TLS handshake error from ***:45385: EOF
I0306 07:32:15.253877       1 logs.go:40] http: TLS handshake error from ***:37527: EOF
I0306 07:32:15.265899       1 logs.go:40] http: TLS handshake error from ***:57258: EOF
I0306 07:32:15.272564       1 logs.go:40] http: TLS handshake error from ***:57249: EOF
I0306 07:32:15.282808       1 logs.go:40] http: TLS handshake error from ***:59928: EOF

主节点中的

dmesg：

[Sun Mar  6 07:32:04 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:04 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:04 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:04 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:04 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:15 2016] net_ratelimit: 34 callbacks suppressed
[Sun Mar  6 07:32:15 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:18 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:18 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:18 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:21 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:21 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:21 2016] TCP: too many orphaned sockets
[Sun Mar  6 07:32:29 2016] TCP: too many orphaned sockets

Answer 1

经过4个小时的调查，结果是因为我的应用程序查询了k8s api服务器。它是用Golang编写的，并使用“gorequest”库向api服务器调用REST api。

gorequest发送后没有关闭请求，即使我在代码中明确地关闭了它。并且很难检查打开连接的数量，因为它在Docker容器内运行。通常通过命令ls /proc/PID/fd | wc -l检查主机是足够的，但这次，我必须访问容器内部进行检查。所以我尝试使用“http”库而不是gorequest，它解决了这个问题！

Kubernetes api服务器从pod中删除请求导致拨号错误

1 个答案: