Kubernetes有大量处于错误状态的pod似乎无法清除

时间:2017-06-06 00:51:38

标签: error-handling kubernetes kubectl

我原本试图运行一个似乎陷入CrashBackoffLoop的Job。这是服务文件:

apiVersion: batch/v1
kind: Job
metadata:
  name: es-setup-indexes
  namespace: elk-test
spec:
  template:
    metadata:
      name: es-setup-indexes
    spec:
      containers:
      - name: es-setup-indexes
        image: appropriate/curl
        command: ['curl -H  "Content-Type: application/json" -XPUT http://elasticsearch.elk-test.svc.cluster.local:9200/_template/filebeat -d@/etc/filebeat/filebeat.template.json']
        volumeMounts:
        - name: configmap-volume
          mountPath: /etc/filebeat/filebeat.template.json
          subPath: filebeat.template.json
      restartPolicy: Never

      volumes:
        - name: configmap-volume
          configMap:
            name: elasticsearch-configmap-indexes

我尝试删除该作业,但只有在运行以下命令时才会起作用:

kubectl delete job es-setup-indexes --cascade=false

之后我发现跑步时注意到了:

kubectl get pods -w

我会得到一个处于错误状态的TON的pod,我认为没办法清理它们。以下是运行get pods时输出的一小部分示例:

es-setup-indexes-zvx9c   0/1       Error     0         20h
es-setup-indexes-zw23w   0/1       Error     0         15h
es-setup-indexes-zw57h   0/1       Error     0         21h
es-setup-indexes-zw6l9   0/1       Error     0         16h
es-setup-indexes-zw7fc   0/1       Error     0         22h
es-setup-indexes-zw9bw   0/1       Error     0         12h
es-setup-indexes-zw9ck   0/1       Error     0         1d
es-setup-indexes-zwf54   0/1       Error     0         18h
es-setup-indexes-zwlmg   0/1       Error     0         16h
es-setup-indexes-zwmsm   0/1       Error     0         21h
es-setup-indexes-zwp37   0/1       Error     0         22h
es-setup-indexes-zwzln   0/1       Error     0         22h
es-setup-indexes-zx4g3   0/1       Error     0         11h
es-setup-indexes-zx4hd   0/1       Error     0         21h
es-setup-indexes-zx512   0/1       Error     0         1d
es-setup-indexes-zx638   0/1       Error     0         17h
es-setup-indexes-zx64c   0/1       Error     0         21h
es-setup-indexes-zxczt   0/1       Error     0         15h
es-setup-indexes-zxdzf   0/1       Error     0         14h
es-setup-indexes-zxf56   0/1       Error     0         1d
es-setup-indexes-zxf9r   0/1       Error     0         16h
es-setup-indexes-zxg0m   0/1       Error     0         14h
es-setup-indexes-zxg71   0/1       Error     0         1d
es-setup-indexes-zxgwz   0/1       Error     0         19h
es-setup-indexes-zxkpm   0/1       Error     0         23h
es-setup-indexes-zxkvb   0/1       Error     0         15h
es-setup-indexes-zxpgg   0/1       Error     0         20h
es-setup-indexes-zxqh3   0/1       Error     0         1d
es-setup-indexes-zxr7f   0/1       Error     0         22h
es-setup-indexes-zxxbs   0/1       Error     0         13h
es-setup-indexes-zz7xr   0/1       Error     0         12h
es-setup-indexes-zzbjq   0/1       Error     0         13h
es-setup-indexes-zzc0z   0/1       Error     0         16h
es-setup-indexes-zzdb6   0/1       Error     0         1d
es-setup-indexes-zzjh2   0/1       Error     0         21h
es-setup-indexes-zzm77   0/1       Error     0         1d
es-setup-indexes-zzqt5   0/1       Error     0         12h
es-setup-indexes-zzr79   0/1       Error     0         16h
es-setup-indexes-zzsfx   0/1       Error     0         1d
es-setup-indexes-zzx1r   0/1       Error     0         21h
es-setup-indexes-zzx6j   0/1       Error     0         1d
kibana-kq51v   1/1       Running   0         10h

但是,如果我看看这些工作,我就不会再与此相关了:

$ kubectl get jobs --all-namespaces                                                                              
NAMESPACE     NAME               DESIRED   SUCCESSFUL   AGE
kube-system   configure-calico   1         1            46d

我也注意到kubectl的反应似乎很慢。我不知道pod是不是一直在尝试重新启动或处于某种破坏状态,但如果有人能让我知道如何排除故障会很好,因为我没有在kubernetes中遇到过这样的问题。

Kube信息:

$ kubectl version 
Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"b0b7a323cc5a4a2019b2e9520c21c7830b7f708e", GitTreeState:"clean", BuildDate:"2017-04-03T20:33:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}

4 个答案:

答案 0 :(得分:14)

在这里,您可以快速解决问题:)

kubectl get pods -a | grep Error | cut -d' ' -f 1 | xargs kubectl delete pod

答案 1 :(得分:7)

Vector3 stairSize; float stepWidthFactor=1f; Vector3 stairsStartPosition; private IEnumerator BuildStairs() { for (int i = 0; i <= stairsNumber ; i++) { stairsPosition = new Vector3( stairsStartPosition.x, stairsStartPosition.y + stairsHeight, stairsStartPosition.z + stairsHeight*stepWidthFactor); GameObject stairs = Instantiate( stairsPrefab, stairsPosition, Quaternion.identity); stairsHeight += stairsSize.y; stairs.transform.localScale = stairSize; yield return new WaitForSeconds(delay); } }

...清理您的命名空间中所有失败的Pod。

答案 2 :(得分:4)

我通常会使用此命令删除所有Error窗格。 kubectl delete pod `kubectl get pods --namespace <yournamespace> | awk '$3 == "Error" {print $1}'` --namespace <yournamespace>

答案 3 :(得分:0)

解决方案是评论中提到的@ johnharris85。我不得不手动删除所有的pod。为此,我运行了以下内容:

kubectl get pods -w | tee all-pods.txt

抛弃我的所有吊舱,然后过滤和删除我想要的东西。

kubectl delete pod $(more all-pods.txt | grep es-setup-index | awk '{print $1}')

注意: 我有大约9292个播客,大约需要1-2个小时来删除它们。