kube-apiserver nonfunctional,"名称命名空间的索引不存在"

时间:2017-02-26 04:05:09

标签: kubernetes

我使用kubeadm在两个节点上启动了Kubernetes 1.5.3集群(详情如下)。我今晚一直试图启动prometheus-operator。删除kind = Prometheus的资源后,我的群集进入了一个非常奇怪的状态,当我尝试重新创建prometheus资源时,虽然存在TPR,但没有创建任何pod。调查引导我:

rrix@hypervisor01:~$ kubectl logs -n kube-system kube-apiserver-hypervisor01 | tail -10 W0226 03:57:34.641244 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:57:34.645073 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:04.642150 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:04.647953 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:34.642118 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:58:34.646427 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:04.642189 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:04.647978 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:34.646129 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist W0226 03:59:34.666355 1 listers.go:69] can not retrieve list of objects using index : Index with name namespace does not exist

rancher github issue中的解决方法是简单地重新启动apiserver,这没关系,除非这并没有解决问题。否则,它指向etcd上的高负载,如下所示,在主机上托管为当前负载小于1的主机.etdctl命令成功运行,让我相信etcd本身很好。为了好的措施,我碰到了etcd docker容器,它仍然可以正常工作。但是,从这里开始,我很遗憾,需要一些帮助才能使我的集群恢复正常工作。

群集概述:

rrix@hypervisor01:~$ kubectl get no NAME STATUS AGE hypervisor01 Ready,master 2d kubes01.pss9.kickass.systems Ready 2d rrix@hypervisor01:~$ kubectl get -n kube-system po NAME READY STATUS RESTARTS AGE dummy-2088944543-2p3zf 1/1 Running 1 2d etcd-hypervisor01 1/1 Running 2 2d kube-apiserver-hypervisor01 1/1 Running 3 2d kube-controller-manager-hypervisor01 1/1 Running 1 2d kube-discovery-1769846148-v8h50 1/1 Running 1 2d kube-dns-2924299975-3s26d 4/4 Running 4 2d kube-proxy-vpw73 1/1 Running 1 2d kube-proxy-zfh13 1/1 Running 0 2d kube-registry-proxy-6hhk9 1/1 Running 1 2d kube-registry-proxy-nl1s1 1/1 Running 0 2d kube-registry-v0-4d94t 1/1 Running 0 2d kube-scheduler-hypervisor01 1/1 Running 1 2d rrix@hypervisor01:~$ kubectl get po NAME READY STATUS RESTARTS AGE kube-flannel-ds-589fw 2/2 Running 3 2d kube-flannel-ds-7f5sx 2/2 Running 0 2d kube-state-metrics-3229993571-20nbc 1/1 Running 0 1h node-exporter-1cdlj 1/1 Running 0 1h node-exporter-jc54s 1/1 Running 0 1h prometheus-operator-996254120-0wzg6 1/1 Running 0 2h

1 个答案:

答案 0 :(得分:1)

E0226 16:20:47.861762 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-k8s, requeuing: Failed to create prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value I0226 16:20:47.862793 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-k8s", UID:"272d42fd-fbd1-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304212", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0, error: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value E0226 16:20:47.865352 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-services, requeuing: Failed to create prometheus-prometheus-services-db-prometheus-prometheus-services-0: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value I0226 16:20:47.865472 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-services", UID:"1899b833-fbd2-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304733", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-services-db-prometheus-prometheus-services-0, error: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value E0226 16:21:17.854692 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-k8s, requeuing: Failed to create prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value I0226 16:21:17.855043 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-k8s", UID:"272d42fd-fbd1-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304212", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0, error: PersistentVolumeClaim "prometheus-prometheus-k8s-db-prometheus-prometheus-k8s-0" is invalid: spec.resources[storage]: Required value E0226 16:21:17.858436 1 pet_set.go:272] Error syncing StatefulSet default/prometheus-prometheus-services, requeuing: Failed to create prometheus-prometheus-services-db-prometheus-prometheus-services-0: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value I0226 16:21:17.858561 1 event.go:217] Event(api.ObjectReference{Kind:"StatefulSet", Namespace:"default", Name:"prometheus-prometheus-services", UID:"1899b833-fbd2-11e6-9ae6-a0481cb808c8", APIVersion:"apps", ResourceVersion:"304733", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' pvc: prometheus-prometheus-services-db-prometheus-prometheus-services-0, error: PersistentVolumeClaim "prometheus-prometheus-services-db-prometheus-prometheus-services-0" is invalid: spec.resources[storage]: Required value

用户错误 facepalm ,没想到查看管理器日志。