Kubelet无法更新节点状态

时间:2016-07-04 15:39:04

标签: kubernetes

在最新的RHEL Atomic Host(Kubernetes 1.2)上,我们经常在kubelet日志中看到以下条目:

kubelet.go:2761]更新节点状态时出错,将重试:节点“x.y.z”无法更新:对象已被修改;请将您的更改应用到最新版本,然后重试

这会导致节点临时转到NotReady。在这些NotReady期间,节点上的POD显示Ready,但看起来Kubernetes停止向它们路由流量,导致我们出现问题。

在Go源代码中我可以看到,在Kubelet的心跳期间,它执行GET以获取最新状态,用自己的状态覆盖它,并将PUT发送回apiserver。

这就是我们在日志中看到的内容:

Jul 15 12:42:45 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:42:45.086322 3736 round_trippers.go:264] GET https://lxa160g.srv.pl.ing.net:6443/api/v1/nodes/lxa160j.srv.pl.ing.net Jul 15 12:42:45 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:42:45.091579 3736 round_trippers.go:289] Response Status: 200 OK in 5 milliseconds Jul 15 12:42:45 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:42:45.373091 3736 round_trippers.go:264] PUT https://lxa160g.srv.pl.ing.net:6443/api/v1/nodes/lxa160j.srv.pl.ing.net/status Jul 15 12:42:45 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:42:45.409752 3736 round_trippers.go:289] Response Status: 200 OK in 36 milliseconds Jul 15 12:42:55 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:42:55.411267 3736 round_trippers.go:264] GET https://lxa160g.srv.pl.ing.net:6443/api/v1/nodes/lxa160j.srv.pl.ing.net Jul 15 12:42:55 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:42:55.431056 3736 round_trippers.go:289] Response Status: 200 OK in 19 milliseconds Jul 15 12:43:38 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:43:38.020203 3736 round_trippers.go:264] PUT https://lxa160g.srv.pl.ing.net:6443/api/v1/nodes/lxa160j.srv.pl.ing.net/status Jul 15 12:43:38 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:43:38.029575 3736 round_trippers.go:289] Response Status: 409 Conflict in 9 milliseconds Jul 15 12:43:38 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:43:38.029772 3736 round_trippers.go:264] GET https://lxa160g.srv.pl.ing.net:6443/api/v1/nodes/lxa160j.srv.pl.ing.net Jul 15 12:43:38 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:43:38.034980 3736 round_trippers.go:289] Response Status: 200 OK in 5 milliseconds Jul 15 12:43:38 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:43:38.298752 3736 round_trippers.go:264] PUT https://lxa160g.srv.pl.ing.net:6443/api/v1/nodes/lxa160j.srv.pl.ing.net/status Jul 15 12:43:38 lxa160j.srv.pl.ing.net kubelet[3736]: I0715 12:43:38.320192 3736 round_trippers.go:289] Response Status: 200 OK in 21 milliseconds

因此,成功获取GET后需要很长时间才能触发PUT。为什么呢?

由于

0 个答案:

没有答案