Question

我想要一个从0扩展到x pod的实例组。我得到Insufficient nvidia.com/gpu。有人看到我在这里做错了吗？这是在Kubernetes v1.9.6上使用自动缩放器1.1.2。

我有两个实例组，一个是cpus，另一个是我想缩小到0个节点，称为gpus，所以kops edit ig gpus是：

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2018-05-31T09:27:31Z
  labels:
    kops.k8s.io/cluster: ci.k8s.local
  name: gpus
spec:
  cloudLabels:
    instancegroup: gpus
    k8s.io/cluster-autoscaler/enabled: ""
  image: ami-4450543d
  kubelet:
    featureGates:
      DevicePlugins: "true"
  machineType: p2.xlarge
  maxPrice: "0.5"
  maxSize: 3
  minSize: 0
  nodeLabels:
    kops.k8s.io/instancegroup: gpus
  role: Node
  rootVolumeOptimization: true
  subnets:
  - eu-west-1c

自动调节器部署有：

    spec:
      containers:
      - command:
        - ./cluster-autoscaler
        - --v=4
        - --stderrthreshold=info
        - --cloud-provider=aws
        - --skip-nodes-with-local-storage=false
        - --nodes=0:3:gpus.ci.k8s.local
        env:
        - name: AWS_REGION
          value: eu-west-1
        image: k8s.gcr.io/cluster-autoscaler:v1.1.2

现在我尝试部署一个简单的GPU测试：

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: simple-gpu-test
spec: 
  replicas: 1
  template:
    metadata:
      labels:
        app: "simplegputest"
    spec:
      containers: 
      - name: "nvidia-smi-gpu"
        image: "nvidia/cuda:8.0-cudnn5-runtime"
        resources: 
          limits: 
             nvidia.com/gpu: 1 # requesting 1 GPU
        volumeMounts:
        - mountPath: /usr/local/nvidia
          name: nvidia
        command: [ "/bin/bash", "-c", "--" ]
        args: [ "while true; do nvidia-smi; sleep 5; done;" ]
      volumes:
      - hostPath:
          path: /usr/local/nvidia
        name: nvidia

我希望实例组从0变为1，但自动缩放器日志显示：

I0605 11:27:29.865576       1 scale_up.go:54] Pod default/simple-gpu-test-6f48d9555d-l9822 is unschedulable
I0605 11:27:29.961051       1 scale_up.go:86] Upcoming 0 nodes
I0605 11:27:30.005163       1 scale_up.go:146] Scale-up predicate failed: PodFitsResources predicate mismatch, cannot put default/simple-gpu-test-6f48d9555d-l9822 on template-node-for-gpus.ci.k8s.local-5829202798403814789, reason: Insufficient nvidia.com/gpu
I0605 11:27:30.005262       1 scale_up.go:175] No pod can fit to gpus.ci.k8s.local
I0605 11:27:30.005324       1 scale_up.go:180] No expansion options
I0605 11:27:30.005393       1 static_autoscaler.go:299] Calculating unneeded nodes
I0605 11:27:30.008919       1 factory.go:33] Event(v1.ObjectReference{Kind:"Pod", Namespace:"default", Name:"simple-gpu-test-6f48d9555d-l9822", UID:"3416d787-68b3-11e8-8e8f-0639a6e973b0", APIVersion:"v1", ResourceVersion:"12429157", FieldPath:""}): type: 'Normal' reason: 'NotTriggerScaleUp' pod didn't trigger scale-up (it wouldn't fit if a new node is added)
I0605 11:27:30.031707       1 leaderelection.go:199] successfully renewed lease kube-system/cluster-autoscaler

当我通过设置最小值1来启动节点时，我看到它具有容量：

Capacity:
 cpu:             4
 memory:          62884036Ki
 nvidia.com/gpu:  1
 pods:            110

and labels
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=p2.xlarge
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=eu-west-1
                    failure-domain.beta.kubernetes.io/zone=eu-west-1c
                    kops.k8s.io/instancegroup=gpus
                    kubernetes.io/role=node
                    node-role.kubernetes.io/node=
                    spot=true

AWS Scale组上存在所需的标记：

{
    "ResourceId": "gpus.ci.k8s.local",
    "ResourceType": "auto-scaling-group",
    "Key": "k8s.io/cluster-autoscaler/node-template/label/kops.k8s.io/instancegroup",
    "Value": "gpus",
    "PropagateAtLaunch": true
}

最后，当我将最小池大小设置为1时，它可以自动从1缩放到3。只是不做0到1。

有没有我可以检查模板以查看它没有资源的原因？

Answer 1

Cluster Autoscaler 是一个独立的程序，可以调整Kubernetes集群的大小以满足当前的需求。 Cluster Autoscaler可以以相同的方式管理云提供商提供的GPU资源。

基于群集autoscaler documentation，对于AWS，可以将节点组缩放为0（显然从0开始），假设满足所有缩小条件。

回到您的问题，对于AWS，如果您使用的是nodeSelector，则需要使用＆＃34; k8s.io/cluster-autoscaler/node-template/label/&等标签在ASG模板中标记您的节点。＃34 ;. 请注意，Kubernetes和AWS GPU支持需要不同的标签。

例如，对于foo = bar的节点标签，您可以使用以下标记来标记ASG：

{
    "ResourceType": "auto-scaling-group",
    "ResourceId": "foo.example.com",
    "PropagateAtLaunch": true,
    "Value": "bar",
    "Key": "k8s.io/cluster-autoscaler/node-template/label/foo"
}

Answer 2

还要确保对AWS账户中所需实例类型的限制为非零。

如何在AWS上使用autoscaler扩展GPU节点

2 个答案: