有没有办法查看pod终止状态的历史记录?例如。如果我查看pod describe
命令,我会看到与此类似的输出:
State: Running
Started: Mon, 10 Jul 2017 13:09:20 +0300
Last State: Terminated
Reason: OOMKilled
Exit Code: 137
Started: Thu, 06 Jul 2017 11:01:21 +0300
Finished: Mon, 10 Jul 2017 13:09:18 +0300
相同的pod describe
在pod事件中没有显示任何内容:
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
10m 10m 1 kubelet, gke-dev-default-d8f2dbc5-mbkb spec.containers{demo} Normal Pulled Container image "eu.gcr.io/project/image:v1" already present on machine
10m 10m 1 kubelet, gke-dev-default-d8f2dbc5-mbkb spec.containers{demo} Normal Created Created container with id 1d857caae77bdc43f0bc90fe045ed5050f85436479073b0e6b46454500f4eb5a
10m 10m 1 kubelet, gke-dev-default-d8f2dbc5-mbkb spec.containers{demo} Normal Started Started container with id 1d857caae77bdc43f0bc90fe045ed5050f85436479073b0e6b46454500f4eb5a
如果我查看kubectl get events --all-namespaces
我看到此事件,但无法将其与特定广告连播相关联:
default 12m 12m 1 gke-dev-default-d8f2dbc5-mbkb Node Warning OOMKilling kernel-monitor, gke-dev-default-d8f2dbc5-mbkb Memory cgroup out of memory: Kill process 1639 (java) score 2014 or sacrifice child
Killed process 1639 (java) total-vm:10828960kB, anon-rss:1013756kB, file-rss:22308kB
甚至通过api重新设置的事件详细信息都有误导性信息(例如default
名称空间虽然pod在demo
名称空间中实际存在):
"metadata": {
"name": "gke-dev-default-d8f2dbc5-mbkb.14cff03fe771b053",
"namespace": "default",
"selfLink": "/api/v1/namespaces/default/events/gke-dev-default-d8f2dbc5-mbkb.14cff03fe771b053",
"uid": "d5d3230e-6557-11e7-a486-42010a8401d3",
"resourceVersion": "5278875",
"creationTimestamp": "2017-07-10T10:09:18Z"
},
"involvedObject": {
"kind": "Node",
"name": "gke-dev-default-d8f2dbc5-mbkb",
"uid": "gke-dev-default-d8f2dbc5-mbkb"
},
"reason": "OOMKilling",
"message": "Memory cgroup out of memory: Kill process 1639 (java) score 2014 or sacrifice child\nKilled process 1639 (java) total-vm:10828960kB, anon-rss:1013756kB, file-rss:22308kB",
"source": {
"component": "kernel-monitor",
"host": "gke-dev-default-d8f2dbc5-mbkb"
},
"firstTimestamp": "2017-07-10T10:09:18Z",
"lastTimestamp": "2017-07-10T10:09:18Z",
"count": 1,
"type": "Warning"
因此,虽然我可以通过pod describe
查看最后的终止状态,但之前的状态如何?
答案 0 :(得分:1)
驱逐事件是节点事件。这就是为什么你在Pod活动中看不到它们的原因。如果您使用运行pod的节点运行kubectl describe node <node_name>
,则可以看到驱逐事件。
测试一下:运行一个不断获得OOMKilled的部署:
kubectl run memory-hog --image=gisleburt/my-memory-hog --replicas=2 --limits=memory=128m
当广告连播开始投放和死亡后,您可以投放kubectl get events
或使用kubectl describe node <node_name>
,然后您会看到以下事件:
Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message
--------- -------- ----- ---- ------------- -------- ------ -------
2m 2m 1 kernel-monitor, gke-test-default-pool-649c88dd-818j Warning OOMKilling Memory cgroup out of memory: Kill process 7345 (exe) score 50000 or sacrifice child
Killed process 7345 (exe) total-vm:6092kB, anon-rss:64kB, file-rss:112kB
2m 2m 1 kernel-monitor, gke-test-default-pool-649c88dd-818j Warning OOMKilling Memory cgroup out of memory: Kill process 7409 (exe) score 51000 or sacrifice child
Killed process 7409 (exe) total-vm:6092kB, anon-rss:68kB, file-rss:112kB
2m 2m 1 kernel-monitor, gke-test-default-pool-649c88dd-818j Warning OOMKilling Memory cgroup out of memory: Kill process 7495 (exe) score 50000 or sacrifice child
Killed process 7495 (exe) total-vm:6092kB, anon-rss:64kB, file-rss:112kB
2m 2m 1 kernel-monitor, gke-test-default-pool-649c88dd-818j Warning OOMKilling Memory cgroup out of memory: Kill process 7561 (exe) score 49000 or sacrifice child
Killed process 7561 (exe) total-vm:6092kB, anon-rss:60kB, file-rss:112kB
2m 2m 1 kernel-monitor, gke-test-default-pool-649c88dd-818j Warning OOMKilling Memory cgroup out of memory: Kill process 7638 (exe) score 494000 or sacrifice child
Killed process 7638 (exe) total-vm:7536kB, anon-rss:148kB, file-rss:1832kB
2m 2m 1 kernel-monitor, gke-test-default-pool-649c88dd-818j Warning OOMKilling Memory cgroup out of memory: Kill process 7728 (exe) score 49000 or sacrifice child
Killed process 7728 (exe) total-vm:6092kB, anon-rss:60kB, file-rss:112kB
2m 2m 1 kernel-monitor, gke-test-default-pool-649c88dd-818j Warning OOMKilling Memory cgroup out of memory: Kill process 7876 (exe) score 48000 or sacrifice child
Killed process 7876 (exe) total-vm:6092kB, anon-rss:60kB, file-rss:112kB
2m 2m 1 kernel-monitor, gke-test-default-pool-649c88dd-818j Warning OOMKilling Memory cgroup out of memory: Kill process 8013 (exe) score 480000 or sacrifice child
Killed process 8013 (exe) total-vm:15732kB, anon-rss:152kB, file-rss:1768kB
2m 2m 1 kernel-monitor, gke-test-default-pool-649c88dd-818j Warning OOMKilling Memory cgroup out of memory: Kill process 8140 (exe) score 1023000 or sacrifice child
Killed process 8140 (exe) total-vm:24184kB, anon-rss:448kB, file-rss:3704kB
2m 25s 50 kernel-monitor, gke-test-default-pool-649c88dd-818j Warning OOMKilling (events with common reason combined)
答案 1 :(得分:0)
您也可以使用
查看上一个已终止的广告连播的日志$ kubectl logs podname –-previous