在Kubernetes集群中运行一段时间后,Eureka Pod变为挂起状态

时间:2020-03-24 04:05:18

标签: kubernetes

我在kubernetes集群(v1.15.2)中部署了一个eureka Pod,今天该Pod变为挂起状态并且实际状态正在运行。其他服务无法访问eureka,表明Pod状态的eureka图标显示: this pod is in a pending state。这是我的有状态部署Yaml:

{
  "kind": "StatefulSet",
  "apiVersion": "apps/v1beta2",
  "metadata": {
    "name": "eureka",
    "namespace": "dabai-fat",
    "selfLink": "/apis/apps/v1beta2/namespaces/dabai-fat/statefulsets/eureka",
    "uid": "92eefc3d-4601-4ebc-9414-8437f9934461",
    "resourceVersion": "20195760",
    "generation": 21,
    "creationTimestamp": "2020-02-01T16:55:54Z",
    "labels": {
      "app": "eureka"
    }
  },
  "spec": {
    "replicas": 1,
    "selector": {
      "matchLabels": {
        "app": "eureka"
      }
    },
    "template": {
      "metadata": {
        "creationTimestamp": null,
        "labels": {
          "app": "eureka"
        }
      },
      "spec": {
        "containers": [
          {
            "name": "eureka",
            "image": "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0",
            "ports": [
              {
                "name": "server",
                "containerPort": 8761,
                "protocol": "TCP"
              },
              {
                "name": "management",
                "containerPort": 8081,
                "protocol": "TCP"
              }
            ],
            "env": [
              {
                "name": "APP_NAME",
                "value": "eureka"
              },
              {
                "name": "POD_NAME",
                "valueFrom": {
                  "fieldRef": {
                    "apiVersion": "v1",
                    "fieldPath": "metadata.name"
                  }
                }
              },
              {
                "name": "APP_OPTS",
                "value": " --spring.application.name=${APP_NAME} --eureka.instance.hostname=${POD_NAME}.${APP_NAME} --registerWithEureka=true --fetchRegistry=true --eureka.instance.preferIpAddress=false --eureka.client.serviceUrl.defaultZone=http://eureka-0.${APP_NAME}:8761/eureka/"
              },
              {
                "name": "APOLLO_META",
                "valueFrom": {
                  "configMapKeyRef": {
                    "name": "fat-config",
                    "key": "apollo.meta"
                  }
                }
              },
              {
                "name": "ENV",
                "valueFrom": {
                  "configMapKeyRef": {
                    "name": "fat-config",
                    "key": "env"
                  }
                }
              }
            ],
            "resources": {
              "limits": {
                "cpu": "2",
                "memory": "1Gi"
              },
              "requests": {
                "cpu": "2",
                "memory": "1Gi"
              }
            },
            "terminationMessagePath": "/dev/termination-log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "IfNotPresent"
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 10,
        "dnsPolicy": "ClusterFirst",
        "securityContext": {},
        "imagePullSecrets": [
          {
            "name": "regcred"
          }
        ],
        "schedulerName": "default-scheduler"
      }
    },
    "serviceName": "eureka-service",
    "podManagementPolicy": "Parallel",
    "updateStrategy": {
      "type": "RollingUpdate",
      "rollingUpdate": {
        "partition": 0
      }
    },
    "revisionHistoryLimit": 10
  },
  "status": {
    "observedGeneration": 21,
    "replicas": 1,
    "readyReplicas": 1,
    "currentReplicas": 1,
    "updatedReplicas": 1,
    "currentRevision": "eureka-5976977b7d",
    "updateRevision": "eureka-5976977b7d",
    "collisionCount": 0
  }
}

这是暂挂状态窗格的描述输出:

$ kubectl describe pod eureka-0
Name:           eureka-0
Namespace:      dabai-fat
Priority:       0
Node:           uat-k8s-01/172.19.104.233
Start Time:     Mon, 23 Mar 2020 18:40:11 +0800
Labels:         app=eureka
                controller-revision-hash=eureka-5976977b7d
                statefulset.kubernetes.io/pod-name=eureka-0
Annotations:    <none>
Status:         Running
IP:             172.30.248.8
IPs:            <none>
Controlled By:  StatefulSet/eureka
Containers:
  eureka:
    Container ID:   docker://5e5eea624e1facc9437fef739669ffeaaa5a7ab655a1297c4acb1e4fd00701ea
    Image:          registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0
    Image ID:       docker-pullable://registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka@sha256:7cd4878ae8efec32984a2b9eec623484c66ae11b9449f8306017cadefbf626ca
    Ports:          8761/TCP, 8081/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Mon, 23 Mar 2020 18:40:18 +0800
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     2
      memory:  1Gi
    Environment:
      APP_NAME:     eureka
      POD_NAME:     eureka-0 (v1:metadata.name)
      APP_OPTS:      --spring.application.name=${APP_NAME} --eureka.instance.hostname=${POD_NAME}.${APP_NAME} --registerWithEureka=true --fetchRegistry=true --eureka.instance.preferIpAddress=false --eureka.client.serviceUrl.defaultZone=http://eureka-0.${APP_NAME}:8761/eureka/
      APOLLO_META:  <set to the key 'apollo.meta' of config map 'fat-config'>  Optional: false
      ENV:          <set to the key 'env' of config map 'fat-config'>          Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xnrwt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-xnrwt:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xnrwt
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 360s
                 node.kubernetes.io/unreachable:NoExecute for 360s
Events:
  Type    Reason     Age   From                 Message
  ----    ------     ----  ----                 -------
  Normal  Scheduled  16h   default-scheduler    Successfully assigned dabai-fat/eureka-0 to uat-k8s-01
  Normal  Pulling    16h   kubelet, uat-k8s-01  Pulling image "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0"
  Normal  Pulled     16h   kubelet, uat-k8s-01  Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0"
  Normal  Created    16h   kubelet, uat-k8s-01  Created container eureka
  Normal  Started    16h   kubelet, uat-k8s-01  Started container eureka

这怎么可能发生?为避免这种情况该怎么办?重新启动eureka pod之后,此问题消失了,但是我仍然想知道导致此问题的原因。

1 个答案:

答案 0 :(得分:1)

听起来像Kubernetes的bug?尝试在当前版本的Kubernetes上重现它。您也可以深入研究kubelet日志,看看是否有任何有用的东西。