Question

我无法创建管道。我什至无法在AI Platform Pipelines仪表板上加载示例/教程，因为它似乎无法代理所需的内容。

An error occurred
Error occured while trying to proxy to: ...

我查看了集群的详细信息，发现了3个有错误的组件：

Deployment  metadata-grpc-deployment     Does not have minimum availability 
Deployment  ml-pipeline  Does not have minimum availability 
Deployment  ml-pipeline-persistenceagent     Does not have minimum availability

创建集群涉及大约在GCP Kubernetes Engine中单击3次，因此我认为我不会搞砸这一步。

任何人都知道如何实现“最低可用性”吗？

更新1

节点具有足够的资源并处于就绪状态。 YAML文件看起来不错。我在差异区域/区域中有2个集群，并且都有上面列出的部署错误。 2个豆荚不好吃。

Name:         ml-pipeline-65479485c8-mcj9x
Namespace:    default
Priority:     0
Node:         gke-cluster-3-default-pool-007784cb-qcsn/10.150.0.2
Start Time:   Thu, 17 Sep 2020 22:15:19 +0000
Labels:       app=ml-pipeline
              app.kubernetes.io/name=kubeflow-pipelines-3
              pod-template-hash=65479485c8
Annotations:  kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container ml-pipeline-api-server

Status:       Running
IP:           10.4.0.8
IPs:
IP:           10.4.0.8
Controlled By:  ReplicaSet/ml-pipeline-65479485c8
Containers:
  ml-pipeline-api-server:
    Container ID:   ...
    Image:          ...
    Image ID:       ...
    Ports:          8888/TCP, 8887/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Fri, 18 Sep 2020 10:27:31 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Fri, 18 Sep 2020 10:20:38 +0000
      Finished:     Fri, 18 Sep 2020 10:27:31 +0000
    Ready:          False
    Restart Count:  98
    Requests:
      cpu:      100m
    Liveness:   exec [wget -q -S -O - http://localhost:8888/apis/v1beta1/healthz] delay=3s timeout=2s period=5s #success=1 #failure=3
    Readiness:  exec [wget -q -S -O - http://localhost:8888/apis/v1beta1/healthz] delay=3s timeout=2s period=5s #success=1 #failure=3
    Environment:
      HAS_DEFAULT_BUCKET:                   true
      BUCKET_NAME:
      PROJECT_ID:                           <set to the key 'project_id' of config map 'gcp-default-config'>  Optional: false
      POD_NAMESPACE:                        default (v1:metadata.namespace)
      DEFAULTPIPELINERUNNERSERVICEACCOUNT:  pipeline-runner
      OBJECTSTORECONFIG_SECURE:             false
      OBJECTSTORECONFIG_BUCKETNAME:
      DBCONFIG_DBNAME:                      kubeflow_pipelines_3_pipeline
      DBCONFIG_USER:                        <set to the key 'username' in secret 'mysql-credential'>  Optional: false
      DBCONFIG_PASSWORD:                    <set to the key 'password' in secret 'mysql-credential'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from ml-pipeline-token-77xl8 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  ml-pipeline-token-77xl8:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ml-pipeline-token-77xl8
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From                                               Message
  ----     ------     ----                  ----                                               -------
  Warning  BackOff    52m (x409 over 11h)   kubelet, gke-cluster-3-default-pool-007784cb-qcsn  Back-off restarting failed container
  Warning  Unhealthy  31m (x94 over 12h)    kubelet, gke-cluster-3-default-pool-007784cb-qcsn  Readiness probe failed:
  Warning  Unhealthy  31m (x29 over 10h)    kubelet, gke-cluster-3-default-pool-007784cb-qcsn  (combined from similar events): Readiness probe failed: c
annot exec in a stopped state: unknown
  Warning  Unhealthy  17m (x95 over 12h)    kubelet, gke-cluster-3-default-pool-007784cb-qcsn  Liveness probe failed:
  Normal   Pulled     7m26s (x97 over 12h)  kubelet, gke-cluster-3-default-pool-007784cb-qcsn  Container image "gcr.io/cloud-marketplace/google-cloud-ai
-platform/kubeflow-pipelines/apiserver:1.0.0" already present on machine
  Warning  Unhealthy  75s (x78 over 12h)    kubelet, gke-cluster-3-default-pool-007784cb-qcsn  Liveness probe errored: rpc error: code = DeadlineExceede
d desc = context deadline exceeded

另一个Pod：

Name:         ml-pipeline-persistenceagent-67db8b8964-mlbmv
Events:
  Type     Reason   Age                   From                                               Message
  ----     ------   ----                  ----                                               -------
  Warning  BackOff  32s (x2238 over 12h)  kubelet, gke-cluster-3-default-pool-007784cb-qcsn  Back-off restarting failed container

解决方案

请勿让Google处理任何存储空间。取消选中“使用托管存储”，然后手动设置您自己的工件集合。您实际上不需要在这些字段中输入任何内容，因为管道仍会启动。

Answer 1

Does not have minimum availability错误是通用的。可能有许多问题触发了它。您需要进行更深入的分析才能找到实际的问题。以下是一些可能的原因：

资源不足：：检查您的节点是否有足够的资源（CPU /内存）。如果Node比check the Pod's status还好。
“活力”探测和/或“就绪”探测失败：执行kubectl describe pod <pod-name>来检查它们是否失败以及原因。
部署错误配置：：查看您的部署yaml文件，以查看以前的配置是否存在任何错误或遗漏。
您还可以尝试稍等一下，因为有时需要一些时间才能部署所有内容和/或尝试更改区域/区域。

GCP AI平台-管道-群集-没有最低可用性

1 个答案: