我在kubernetes集群(v1.15.2)中部署了一个eureka Pod,今天该Pod变为挂起状态并且实际状态正在运行。其他服务无法访问eureka,表明Pod状态的eureka图标显示: this pod is in a pending state
。这是我的有状态部署Yaml:
{
"kind": "StatefulSet",
"apiVersion": "apps/v1beta2",
"metadata": {
"name": "eureka",
"namespace": "dabai-fat",
"selfLink": "/apis/apps/v1beta2/namespaces/dabai-fat/statefulsets/eureka",
"uid": "92eefc3d-4601-4ebc-9414-8437f9934461",
"resourceVersion": "20195760",
"generation": 21,
"creationTimestamp": "2020-02-01T16:55:54Z",
"labels": {
"app": "eureka"
}
},
"spec": {
"replicas": 1,
"selector": {
"matchLabels": {
"app": "eureka"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"app": "eureka"
}
},
"spec": {
"containers": [
{
"name": "eureka",
"image": "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0",
"ports": [
{
"name": "server",
"containerPort": 8761,
"protocol": "TCP"
},
{
"name": "management",
"containerPort": 8081,
"protocol": "TCP"
}
],
"env": [
{
"name": "APP_NAME",
"value": "eureka"
},
{
"name": "POD_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.name"
}
}
},
{
"name": "APP_OPTS",
"value": " --spring.application.name=${APP_NAME} --eureka.instance.hostname=${POD_NAME}.${APP_NAME} --registerWithEureka=true --fetchRegistry=true --eureka.instance.preferIpAddress=false --eureka.client.serviceUrl.defaultZone=http://eureka-0.${APP_NAME}:8761/eureka/"
},
{
"name": "APOLLO_META",
"valueFrom": {
"configMapKeyRef": {
"name": "fat-config",
"key": "apollo.meta"
}
}
},
{
"name": "ENV",
"valueFrom": {
"configMapKeyRef": {
"name": "fat-config",
"key": "env"
}
}
}
],
"resources": {
"limits": {
"cpu": "2",
"memory": "1Gi"
},
"requests": {
"cpu": "2",
"memory": "1Gi"
}
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"imagePullPolicy": "IfNotPresent"
}
],
"restartPolicy": "Always",
"terminationGracePeriodSeconds": 10,
"dnsPolicy": "ClusterFirst",
"securityContext": {},
"imagePullSecrets": [
{
"name": "regcred"
}
],
"schedulerName": "default-scheduler"
}
},
"serviceName": "eureka-service",
"podManagementPolicy": "Parallel",
"updateStrategy": {
"type": "RollingUpdate",
"rollingUpdate": {
"partition": 0
}
},
"revisionHistoryLimit": 10
},
"status": {
"observedGeneration": 21,
"replicas": 1,
"readyReplicas": 1,
"currentReplicas": 1,
"updatedReplicas": 1,
"currentRevision": "eureka-5976977b7d",
"updateRevision": "eureka-5976977b7d",
"collisionCount": 0
}
}
这是暂挂状态窗格的描述输出:
$ kubectl describe pod eureka-0
Name: eureka-0
Namespace: dabai-fat
Priority: 0
Node: uat-k8s-01/172.19.104.233
Start Time: Mon, 23 Mar 2020 18:40:11 +0800
Labels: app=eureka
controller-revision-hash=eureka-5976977b7d
statefulset.kubernetes.io/pod-name=eureka-0
Annotations: <none>
Status: Running
IP: 172.30.248.8
IPs: <none>
Controlled By: StatefulSet/eureka
Containers:
eureka:
Container ID: docker://5e5eea624e1facc9437fef739669ffeaaa5a7ab655a1297c4acb1e4fd00701ea
Image: registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0
Image ID: docker-pullable://registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka@sha256:7cd4878ae8efec32984a2b9eec623484c66ae11b9449f8306017cadefbf626ca
Ports: 8761/TCP, 8081/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Mon, 23 Mar 2020 18:40:18 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 2
memory: 1Gi
Environment:
APP_NAME: eureka
POD_NAME: eureka-0 (v1:metadata.name)
APP_OPTS: --spring.application.name=${APP_NAME} --eureka.instance.hostname=${POD_NAME}.${APP_NAME} --registerWithEureka=true --fetchRegistry=true --eureka.instance.preferIpAddress=false --eureka.client.serviceUrl.defaultZone=http://eureka-0.${APP_NAME}:8761/eureka/
APOLLO_META: <set to the key 'apollo.meta' of config map 'fat-config'> Optional: false
ENV: <set to the key 'env' of config map 'fat-config'> Optional: false
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-xnrwt (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady True
PodScheduled True
Volumes:
default-token-xnrwt:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-xnrwt
Optional: false
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 360s
node.kubernetes.io/unreachable:NoExecute for 360s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 16h default-scheduler Successfully assigned dabai-fat/eureka-0 to uat-k8s-01
Normal Pulling 16h kubelet, uat-k8s-01 Pulling image "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0"
Normal Pulled 16h kubelet, uat-k8s-01 Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/dabai_app_k8s/dabai_fat/soa-eureka:v1.0.0"
Normal Created 16h kubelet, uat-k8s-01 Created container eureka
Normal Started 16h kubelet, uat-k8s-01 Started container eureka
这怎么可能发生?为避免这种情况该怎么办?重新启动eureka pod之后,此问题消失了,但是我仍然想知道导致此问题的原因。
答案 0 :(得分:1)
听起来像Kubernetes的bug?尝试在当前版本的Kubernetes上重现它。您也可以深入研究kubelet日志,看看是否有任何有用的东西。