Kubernetes Pod终止活动的历史?

时间:2017-07-10 10:38:32

标签: kubernetes

有没有办法查看pod终止状态的历史记录?例如。如果我查看pod describe命令,我会看到与此类似的输出:

State:      Running
  Started:      Mon, 10 Jul 2017 13:09:20 +0300
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
  Started:      Thu, 06 Jul 2017 11:01:21 +0300
  Finished:     Mon, 10 Jul 2017 13:09:18 +0300

相同的pod describe在pod事件中没有显示任何内容:

   Events:
  FirstSeen LastSeen    Count   From                    SubObjectPath       Type        Reason  Message
  --------- --------    -----   ----                    -------------       --------    ------  -------
  10m       10m     1   kubelet, gke-dev-default-d8f2dbc5-mbkb  spec.containers{demo}   Normal      Pulled  Container image "eu.gcr.io/project/image:v1" already present on machine
  10m       10m     1   kubelet, gke-dev-default-d8f2dbc5-mbkb  spec.containers{demo}   Normal      Created Created container with id 1d857caae77bdc43f0bc90fe045ed5050f85436479073b0e6b46454500f4eb5a
  10m       10m     1   kubelet, gke-dev-default-d8f2dbc5-mbkb  spec.containers{demo}   Normal      Started Started container with id 1d857caae77bdc43f0bc90fe045ed5050f85436479073b0e6b46454500f4eb5a

如果我查看kubectl get events --all-namespaces我看到此事件,但无法将其与特定广告连播相关联:

  default   12m       12m       1         gke-dev-default-d8f2dbc5-mbkb   Node                Warning   OOMKilling   kernel-monitor, gke-dev-default-d8f2dbc5-mbkb   Memory cgroup out of memory: Kill process 1639 (java) score 2014 or sacrifice child
Killed process 1639 (java) total-vm:10828960kB, anon-rss:1013756kB, file-rss:22308kB

甚至通过api重新设置的事件详细信息都有误导性信息(例如default名称空间虽然pod在demo名称空间中实际存在):

    "metadata": {
        "name": "gke-dev-default-d8f2dbc5-mbkb.14cff03fe771b053",
        "namespace": "default",
        "selfLink": "/api/v1/namespaces/default/events/gke-dev-default-d8f2dbc5-mbkb.14cff03fe771b053",
        "uid": "d5d3230e-6557-11e7-a486-42010a8401d3",
        "resourceVersion": "5278875",
        "creationTimestamp": "2017-07-10T10:09:18Z"
    },
    "involvedObject": {
        "kind": "Node",
        "name": "gke-dev-default-d8f2dbc5-mbkb",
        "uid": "gke-dev-default-d8f2dbc5-mbkb"
    },
    "reason": "OOMKilling",
    "message": "Memory cgroup out of memory: Kill process 1639 (java) score 2014 or sacrifice child\nKilled process 1639 (java) total-vm:10828960kB, anon-rss:1013756kB, file-rss:22308kB",
    "source": {
        "component": "kernel-monitor",
        "host": "gke-dev-default-d8f2dbc5-mbkb"
    },
    "firstTimestamp": "2017-07-10T10:09:18Z",
    "lastTimestamp": "2017-07-10T10:09:18Z",
    "count": 1,
    "type": "Warning"

因此,虽然我可以通过pod describe查看最后的终止状态,但之前的状态如何?

2 个答案:

答案 0 :(得分:1)

驱逐事件是节点事件。这就是为什么你在Pod活动中看不到它们的原因。如果您使用运行pod的节点运行kubectl describe node <node_name>,则可以看到驱逐事件。

测试一下:运行一个不断获得OOMKilled的部署:

kubectl run memory-hog --image=gisleburt/my-memory-hog --replicas=2 --limits=memory=128m

当广告连播开始投放和死亡后,您可以投放kubectl get events或使用kubectl describe node <node_name>,然后您会看到以下事件:

Events:
  FirstSeen LastSeen    Count   From                            SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----                            -------------   --------    ------      -------
  2m        2m      1   kernel-monitor, gke-test-default-pool-649c88dd-818j         Warning     OOMKilling  Memory cgroup out of memory: Kill process 7345 (exe) score 50000 or sacrifice child
Killed process 7345 (exe) total-vm:6092kB, anon-rss:64kB, file-rss:112kB
  2m    2m  1   kernel-monitor, gke-test-default-pool-649c88dd-818j     Warning OOMKilling  Memory cgroup out of memory: Kill process 7409 (exe) score 51000 or sacrifice child
Killed process 7409 (exe) total-vm:6092kB, anon-rss:68kB, file-rss:112kB
  2m    2m  1   kernel-monitor, gke-test-default-pool-649c88dd-818j     Warning OOMKilling  Memory cgroup out of memory: Kill process 7495 (exe) score 50000 or sacrifice child
Killed process 7495 (exe) total-vm:6092kB, anon-rss:64kB, file-rss:112kB
  2m    2m  1   kernel-monitor, gke-test-default-pool-649c88dd-818j     Warning OOMKilling  Memory cgroup out of memory: Kill process 7561 (exe) score 49000 or sacrifice child
Killed process 7561 (exe) total-vm:6092kB, anon-rss:60kB, file-rss:112kB
  2m    2m  1   kernel-monitor, gke-test-default-pool-649c88dd-818j     Warning OOMKilling  Memory cgroup out of memory: Kill process 7638 (exe) score 494000 or sacrifice child
Killed process 7638 (exe) total-vm:7536kB, anon-rss:148kB, file-rss:1832kB
  2m    2m  1   kernel-monitor, gke-test-default-pool-649c88dd-818j     Warning OOMKilling  Memory cgroup out of memory: Kill process 7728 (exe) score 49000 or sacrifice child
Killed process 7728 (exe) total-vm:6092kB, anon-rss:60kB, file-rss:112kB
  2m    2m  1   kernel-monitor, gke-test-default-pool-649c88dd-818j     Warning OOMKilling  Memory cgroup out of memory: Kill process 7876 (exe) score 48000 or sacrifice child
Killed process 7876 (exe) total-vm:6092kB, anon-rss:60kB, file-rss:112kB
  2m    2m  1   kernel-monitor, gke-test-default-pool-649c88dd-818j     Warning OOMKilling  Memory cgroup out of memory: Kill process 8013 (exe) score 480000 or sacrifice child
Killed process 8013 (exe) total-vm:15732kB, anon-rss:152kB, file-rss:1768kB
  2m    2m  1   kernel-monitor, gke-test-default-pool-649c88dd-818j     Warning OOMKilling  Memory cgroup out of memory: Kill process 8140 (exe) score 1023000 or sacrifice child
Killed process 8140 (exe) total-vm:24184kB, anon-rss:448kB, file-rss:3704kB
  2m    25s 50  kernel-monitor, gke-test-default-pool-649c88dd-818j     Warning OOMKilling  (events with common reason combined)

答案 1 :(得分:0)

您也可以使用

查看上一个已终止的广告连播的日志
$ kubectl logs podname –-previous