pod将无法启动,因为“没有可用的节点匹配以下所有谓词:: CPU不足”

时间:2017-08-08 16:44:42

标签: kubernetes

我不确定为什么我收到错误No nodes are available that match all of the following predicates:: Insufficient cpu (1)

我不记得设置任何CPU限制。除非这是默认值?

kubectl describe pod wordpress的输出:

Name:       wordpress-114465096-bn4rv
Namespace:  default
Node:       /
Labels:     app=wordpress
        pod-template-hash=114465096
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"wordpress-114465096","uid":"fff460df-7c4c-11e7-b3fd-42010a840026...
        kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container wordpress; cpu request for container cloudsql-proxy; cpu request for container nginx
Status:     Pending
IP:     
Controllers:    ReplicaSet/wordpress-114465096
Containers:
  wordpress:
    Image:  wordpress:latest
    Port:   
    Requests:
      cpu:  100m
    Environment:
      WORDPRESS_HOST:       localhost
      WORDPRESS_DB_USERNAME:    <set to the key 'username' in secret 'cloudsql-db-credentials'> Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ql6k8 (ro)
      /var/www/html from wordpress-persistent-storage (rw)
  cloudsql-proxy:
    Image:  gcr.io/cloudsql-docker/gce-proxy:1.09
    Port:   
    Command:
      /cloud_sql_proxy
      --dir=/cloudsql
      -instances=inspiring-tower-99712:europe-west1:wordpressdb=tcp:3306
      -credential_file=/secrets/cloudsql/credentials.json
    Requests:
      cpu:      100m
    Environment:    <none>
    Mounts:
      /cloudsql from cloudsql (rw)
      /etc/ssl/certs from ssl-certs (rw)
      /secrets/cloudsql from cloudsql-instance-credentials (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ql6k8 (ro)
  nginx:
    Image:  nginx:latest
    Port:   80/TCP
    Requests:
      cpu:      100m
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-ql6k8 (ro)
Conditions:
  Type      Status
  PodScheduled  False 
Volumes:
  wordpress-persistent-storage:
    Type:   GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName: wordpress-disk
    FSType: ext4
    Partition:  0
    ReadOnly:   false
  cloudsql-instance-credentials:
    Type:   Secret (a volume populated by a Secret)
    SecretName: cloudsql-instance-credentials
    Optional:   false
  ssl-certs:
    Type:   HostPath (bare host directory volume)
    Path:   /etc/ssl/certs
  cloudsql:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium: 
  default-token-ql6k8:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-ql6k8
    Optional:   false
QoS Class:  Burstable
Node-Selectors: <none>
Tolerations:    node.alpha.kubernetes.io/notReady=:Exists:NoExecute for 300s
        node.alpha.kubernetes.io/unreachable=:Exists:NoExecute for 300s
Events:
  FirstSeen LastSeen    Count   From            SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  1h        22s     265 default-scheduler           Warning     FailedScheduling    No nodes are available that match all of the following predicates:: Insufficient cpu (1).

kubectl describe node gke-wordpress-default-pool-91c14317-jdlj(群集中的单个节点)的输出:

Name:           gke-wordpress-default-pool-91c14317-jdlj
Role:           
Labels:         beta.kubernetes.io/arch=amd64
                        beta.kubernetes.io/fluentd-ds-ready=true
                        beta.kubernetes.io/instance-type=n1-standard-1
                        beta.kubernetes.io/os=linux
                        cloud.google.com/gke-nodepool=default-pool
                        failure-domain.beta.kubernetes.io/region=europe-west1
                        failure-domain.beta.kubernetes.io/zone=europe-west1-b
                        kubernetes.io/hostname=gke-wordpress-default-pool-91c14317-jdlj
Annotations:        node.alpha.kubernetes.io/ttl=0
                        volumes.kubernetes.io/controller-managed-attach-detach=true
Taints:         <none>
CreationTimestamp:  Fri, 04 Aug 2017 17:44:08 +0100
Phase:          
Conditions:
Type            Status  LastHeartbeatTime           LastTransitionTime          Reason              Message
----            ------  -----------------           ------------------          ------              -------
NetworkUnavailable  False   Fri, 04 Aug 2017 17:44:35 +0100     Fri, 04 Aug 2017 17:44:35 +0100     RouteCreated            RouteController created a route
OutOfDisk       False   Tue, 08 Aug 2017 21:04:47 +0100     Fri, 04 Aug 2017 17:44:08 +0100     KubeletHasSufficientDisk    kubelet has sufficient disk space available
MemoryPressure  False   Tue, 08 Aug 2017 21:04:47 +0100     Fri, 04 Aug 2017 17:44:08 +0100     KubeletHasSufficientMemory  kubelet has sufficient memory available
DiskPressure        False   Tue, 08 Aug 2017 21:04:47 +0100     Fri, 04 Aug 2017 17:44:08 +0100     KubeletHasNoDiskPressure    kubelet has no disk pressure
Ready       True    Tue, 08 Aug 2017 21:04:47 +0100     Fri, 04 Aug 2017 17:44:39 +0100     KubeletReady            kubelet is posting ready status. AppArmor enabled
KernelDeadlock  False   Tue, 08 Aug 2017 21:03:56 +0100     Fri, 04 Aug 2017 17:43:19 +0100     KernelHasNoDeadlock         kernel has no deadlock
Addresses:      10.132.0.3,35.195.163.26,gke-wordpress-default-pool-91c14317-jdlj
Capacity:
cpu:        1
memory: 3794520Ki
pods:       110
Allocatable:
cpu:        1
memory: 3794520Ki
pods:       110
System Info:
Machine ID:         2643dae58dd36381dc5e8ebe124272bc
System UUID:            2643DAE5-8DD3-6381-DC5E-8EBE124272BC
Boot ID:            37002900-44ab-45b1-bbca-04d2b5866683
Kernel Version:     4.4.52+
OS Image:           Container-Optimized OS from Google
Operating System:       linux
Architecture:           amd64
Container Runtime Version:  docker://1.11.2
Kubelet Version:        v1.6.7
Kube-Proxy Version:     v1.6.7
PodCIDR:            10.24.0.0/24
ExternalID:         8419821342083849481
Non-terminated Pods:        (7 in total)
Namespace           Name                                CPU Requests    CPU Limits  Memory Requests Memory Limits
---------           ----                                ------------    ----------  --------------- -------------
kube-system         fluentd-gcp-v2.0-t3rzf                      100m (10%)  0 (0%)      200Mi (5%)  300Mi (8%)
kube-system         heapster-v1.3.0-3440173064-d66jq                138m (13%)  138m (13%)  301456Ki (7%)   301456Ki (7%)
kube-system         kube-dns-1829567597-n6kz6                   260m (26%)  0 (0%)      110Mi (2%)  170Mi (4%)
kube-system         kube-dns-autoscaler-2501648610-88ch6                20m (2%)    0 (0%)      10Mi (0%)   0 (0%)
kube-system         kube-proxy-gke-wordpress-default-pool-91c14317-jdlj     100m (10%)  0 (0%)      0 (0%)      0 (0%)
kube-system         kubernetes-dashboard-490794276-93cn2                100m (10%)  100m (10%)  50Mi (1%)   50Mi (1%)
kube-system         l7-default-backend-3574702981-509zt             10m (1%)    10m (1%)    20Mi (0%)   20Mi (0%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
CPU Requests    CPU Limits  Memory Requests Memory Limits
------------    ----------  --------------- -------------
728m (72%)  248m (24%)  700816Ki (18%)  854416Ki (22%)
Events:     <none>

配置文件(production.yaml):

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: wordpress
  labels:
    app: wordpress
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wordpress
  template:
    metadata:
      labels:
        app: wordpress
    spec:
      terminationGracePeriodSeconds: 30
      containers:
        - image: wordpress:latest
          name: wordpress
          imagePullPolicy: "Always"
          env:
            - name: WORDPRESS_HOST
              value: localhost
            - name: WORDPRESS_DB_USERNAME
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username
          volumeMounts:
            - name: wordpress-persistent-storage
              mountPath: /var/www/html
        - image: nginx:latest
          name: nginx
          ports:
            - containerPort: 80
              name: nginx
        - image: gcr.io/cloudsql-docker/gce-proxy:1.09
          name: cloudsql-proxy
          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=inspiring-tower-99712:europe-west1:wordpressdb=tcp:3306",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: ssl-certs
              mountPath: /etc/ssl/certs
            - name: cloudsql
              mountPath: /cloudsql
      volumes:
        - name: wordpress-persistent-storage
          gcePersistentDisk:
            pdName: wordpress-disk
            fsType: ext4

        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir:

3 个答案:

答案 0 :(得分:7)

除了配置中指定的容器外,还有其他容器在运行。这些默认情况下使用Kubernetes进行配置,并在kube-system命名空间中运行,这些命名空间未显示在默认命名空间中。

您可以按kubectl get pods --all-namespaces查看所有广告连播。

这些额外的容器占据了单个节点的CPU配额的72%......

因此,10%cpu配额的3个容器将超过CPU配额的100%(因为72%+(3 * 10)> 100%) ......

至于为什么72%的容器被分配到其他容器 - 问题在这里提出:Why does a single node cluster only have a small percentage of the cpu quota available?

可能有用的其他资源:How to reduce CPU limits of kubernetes system resources?

但是,通过向群集添加其他节点,我能够以足够的CPU运行容器。此外,高cpu实例似乎在Google Cloud上更有效地分配。

答案 1 :(得分:0)

错误是非常自我解释的,您要求300毫米(您的容器中每个容器100米)的CPU,并且您的节点没有预算来安排它。 (您似乎只有一个节点集群?)

您可以描述节点以查看已安排的CPU时间。

我不确定将这些请求添加到您的部署中的是什么,因为您没有在模板中说明这些要求。可能是资源配额。

答案 2 :(得分:0)

我遇到了同样的错误,但我创建了一个只有一个节点的集群。因此,我使用以下命令使用最小和最大节点数更新了我的集群:

gcloud container clusters update <mycluster-name> --enable-autoscaling --min-nodes=1 --max-nodes=15

下一刻,pod / pods状态从Pending变为Running。