如何使用NodeUnderDiskPressure修复Openshift pod启动失败?

时间:2018-01-16 22:03:18

标签: openshift openshift-origin

我使用

在本地启动openshift
oc cluster up

然后我使用hello-pod.json使用此命令

创建一个pod
oc create -f examples/hello-openshift/hello-pod.json

已创建广告连播但无法启动。 Openshift显示错误:

Reason: Failed Scheduling

Message:  0/1 nodes are available: 1 NodeUnderDiskPressure.

我的硬盘上仍有足够的可用空间。我不知道在哪里寻找其他日志。如何解决问题?

2 个答案:

答案 0 :(得分:4)

在我的情况下,调整node-config.yaml解决了问题:

1)搜索生成的文件node-config.yaml,例如在/var/lib/origin/或您的自定义配置路径下。

2)在编辑器中打开并搜索kubeletArguments并添加您希望的磁盘驱逐警察:

kubeletArguments:
  eviction-hard:
  - memory.available<100Mi
  - nodefs.available<1%
  - nodefs.inodesFree<1%
  - imagefs.available<1%

详细说明可在此处找到:OpenShift Documentation - Default Hard Eviction Thresholds

答案 1 :(得分:2)

基本上我只需要在我的家庭用户目录中恢复docker和kubernetes配置的fileSystem。

$ oc cluster down
$ sudo systemctl stop docker
$ sudo rm -rf /var/lib/docker
$ rm -rf ~/.kube
$ sudo systemctl start docker
$ oc cluster up

DONE! - 之后我就能创建pod。

以下是我在识别相同的NodeUnderDiskPressure时尝试过的其他一些事情,如果这不能解决问题,可能对您有所帮助:

首先,我通过以下方式从kubectl中检索了可用的节点:

$ oc login -u system:admin
$ kubectl get nodes
NAME        STATUS    AGE       VERSION
localhost   Ready     12h       v1.7.6+a08f5eeb62

接下来,我检索了localhost节点的描述:

$ kubectl describe node localhost
Name:           localhost
Role:           
Labels:         beta.kubernetes.io/arch=amd64
            beta.kubernetes.io/os=linux
            kubernetes.io/hostname=localhost
Annotations:        volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:         <none>
CreationTimestamp:  Mon, 05 Mar 2018 20:00:20 -0600
Conditions:
  Type          Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
  ----          ------  -----------------           ------------------          ------              -------
  OutOfDisk         False   Tue, 06 Mar 2018 08:09:03 -0600     Mon, 05 Mar 2018 20:00:20 -0600     KubeletHasSufficientDisk    kubelet has sufficient disk space available
  MemoryPressure    False   Tue, 06 Mar 2018 08:09:03 -0600     Mon, 05 Mar 2018 20:00:20 -0600     KubeletHasSufficientMemory  kubelet has sufficient memory available
  DiskPressure      True    Tue, 06 Mar 2018 08:09:03 -0600     Mon, 05 Mar 2018 20:00:31 -0600     KubeletHasDiskPressure      kubelet has disk pressure
  Ready         True    Tue, 06 Mar 2018 08:09:03 -0600     Mon, 05 Mar 2018 20:00:31 -0600     KubeletReady            kubelet is posting ready status
Addresses:
  InternalIP:   192.168.0.14
  Hostname: localhost
Capacity:
 cpu:       4
 memory:    16311024Ki
 pods:      40
Allocatable:
 cpu:       4
 memory:    16208624Ki
 pods:      40
System Info:
 Machine ID:            6895f77789824d26acef6d0db236319f
 System UUID:           248A664C-33F8-11B2-A85C-FC31558EDC86
 Boot ID:           1a5cc22b-81f1-4b07-b26f-917a7d17936f
 Kernel Version:        4.13.16-100.fc25.x86_64
 OS Image:          CentOS Linux 7 (Core)
 Operating System:      linux
 Architecture:          amd64
 Container Runtime Version: docker://1.12.6
 Kubelet Version:       v1.7.6+a08f5eeb62
 Kube-Proxy Version:        v1.7.6+a08f5eeb62
ExternalID:         localhost
Non-terminated Pods:        (0 in total)
  Namespace         Name        CPU Requests    CPU Limits  Memory Requests Memory Limits
  ---------         ----        ------------    ----------  --------------- -------------
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests  CPU Limits  Memory Requests Memory Limits
  ------------  ----------  --------------- -------------
  0 (0%)    0 (0%)      0 (0%)      0 (0%)
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  12h       8m      2877    kubelet, localhost          Warning     EvictionThresholdMet    Attempting to reclaim imagefs
  11h       3m      136 kubelet, localhost          Warning     ImageGCFailed       (combined from similar events): wanted to free 3113113190 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete 933861786d39 (must be forced) - image is being used by stopped container 82eca7ad6fd6"}, rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete bcccfe5352d3 (must be forced) - image is being used by stopped container 9c4ad3dc4b80"}, rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete b7b0dbc4f785 (must be forced) - image is being used by stopped container d388fa17ff84"}, rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete 0129e5e73319 (cannot be forced) - image has dependent child images"}, rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete 725dcfab7d63 (must be forced) - image is being used by stopped container 9eb3a771aa6f"}, rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete 8ec432b4cda3 (cannot be forced) - image is being used by running container a3fe6da22775"}]

有几点需要注意:

  1. DiskPressure True
  2. 中的条件状态
  3. Events警告:首先我可以看到EvictionThreshold尝试回收imagefs;我还可以看到ImageGCFailed,其中包含有关无法处理的图像的详细信息。
  4. 这是我案例中ImageGCFailed消息的格式化JSON:

    (combined from similar events):wanted to free 3113113190 bytes,
    but freed 0 bytes space with errors in image deletion:[  
       rpc error:   code = 2 desc = Error response from daemon:{  
          "message":"conflict: unable to delete 933861786d39 (must be forced) - image is being used by stopped container 82eca7ad6fd6"
       },
       rpc error:   code = 2 desc = Error response from daemon:{  
          "message":"conflict: unable to delete bcccfe5352d3 (must be forced) - image is being used by stopped container 9c4ad3dc4b80"
       },
       rpc error:   code = 2 desc = Error response from daemon:{  
          "message":"conflict: unable to delete b7b0dbc4f785 (must be forced) - image is being used by stopped container d388fa17ff84"
       },
       rpc error:   code = 2 desc = Error response from daemon:{  
          "message":"conflict: unable to delete 0129e5e73319 (cannot be forced) - image has dependent child images"
       },
       rpc error:   code = 2 desc = Error response from daemon:{  
          "message":"conflict: unable to delete 725dcfab7d63 (must be forced) - image is being used by stopped container 9eb3a771aa6f"
       },
       rpc error:   code = 2 desc = Error response from daemon:{  
          "message":"conflict: unable to delete 8ec432b4cda3 (cannot be forced) - image is being used by running container a3fe6da22775"
       }
    ]
    

    根据此信息:https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/#reclaiming-node-level-resources 现在我调查可用容器并尝试手动删除它们:

    $ docker ps -a
    CONTAINER ID        IMAGE                                         COMMAND                  CREATED             STATUS                      PORTS               NAMES
    a3fe6da22775        openshift/origin:v3.7.1                       "/usr/bin/openshift s"   12 hours ago        Up 12 hours                                     origin
    82eca7ad6fd6        dtf-bpms/nodejs-mongo-persistent-2:4e90f728   "/bin/sh -ic 'npm sta"   3 months ago        Exited (137) 3 months ago                       openshift_s2i-build_nodejs-mongo-persistent-2_dtf-bpms_post-commit_fe89fcfd
    9c4ad3dc4b80        dtf-bpms/nodejs-mongo-persistent-2:4e23c7d5   "/bin/sh -ic 'npm tes"   3 months ago        Exited (137) 3 months ago                       openshift_s2i-build_nodejs-mongo-persistent-2_dtf-bpms_post-commit_de141bcd
    d388fa17ff84        dtf-bpms/nodejs-mongo-persistent-1:439d35ea   "/bin/sh -ic 'npm tes"   3 months ago        Exited (137) 3 months ago                       openshift_s2i-build_nodejs-mongo-persistent-1_dtf-bpms_post-commit_277b19ca
    9eb3a771aa6f        hello-world                                   "/hello"                 3 months ago        Exited (0) 3 months ago                         serene_babbage
    

    现在我将手动删除所有已停止的容器:

    $ docker rm $(docker ps -a -q)
    82eca7ad6fd6
    9c4ad3dc4b80
    d388fa17ff84
    9eb3a771aa6f
    Error response from daemon: You cannot remove a running container a3fe6da22775a559fe94ab0eb5f52d55d9aca6d1f950f107d13243fa029e071f. Stop the container before attempting removal or use -f
    

    在这种情况下,保留openshift容器是可以的。

    $ docker ps -a
    CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS               NAMES
    a3fe6da22775        openshift/origin:v3.7.1   "/usr/bin/openshift s"   12 hours ago        Up 12 hours                             origin
    

    接下来我重新启动openshift和docker并尝试再次创建我的容器并描述localhost节点:

    $ oc cluster down
    $ sudo systemctl restart docker
    $ oc cluster up
    ... (wait for cluster up start)
    $ [CREATE PROJECT AND CONTAINERS]
    $ oc login -u system:admin
    $ kubectl describe node localhost
    ... (node description and header information)
    Events:
      FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
      --------- --------    -----   ----            -------------   --------    ------          -------
      1h        1h      2   kubelet, localhost          Normal      NodeHasSufficientMemory Node localhost status is now: NodeHasSufficientMemory
      1h        1h      2   kubelet, localhost          Normal      NodeHasNoDiskPressure   Node localhost status is now: NodeHasNoDiskPressure
      1h        1h      1   kubelet, localhost          Normal      NodeAllocatableEnforced Updated Node Allocatable limit across pods
      1h        1h      2   kubelet, localhost          Normal      NodeHasSufficientDisk   Node localhost status is now: NodeHasSufficientDisk
      1h        1h      1   kubelet, localhost          Normal      NodeReady       Node localhost status is now: NodeReady
      1h        1h      1   kubelet, localhost          Normal      NodeHasDiskPressure Node localhost status is now: NodeHasDiskPressure
      1h        1h      1   kubelet, localhost          Warning     ImageGCFailed       wanted to free 2934625894 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = 2 desc = Error response from daemon: {"message":"conflict: unable to delete 8ec432b4cda3 (cannot be forced) - image is being used by running container 4bcd2196747c"}
    

    您可以看到我在清理旧的未使用容器后继续看到NodeHasDiskPressure,并且已从Docker事件中释放图像。在这里,下一步是删除旧的DIRTY DOCKER文件系统并从一个新的开始。