Kubernetes Pod错误:创建多个服务

时间:2020-10-09 21:47:35

标签: kubernetes deep-learning google-kubernetes-engine gcloud kubernetes-ingress

我是Kubernetese的新手,如果我的问题含糊其辞,我深表歉意。我尽力详细说明。我通过Kubernetese在Google Cloud上安装了一个pod,其中装有GPU。该GPU负责处理一组任务,比方说对图像进行分类。为此,我使用kubernetes创建了一个服务。我的yaml文件的service部分如下所示。此外,由于该服务的名称为http://model-server-service.default.svc.cluster.local

,因此该服务的网址将为moderl-server-service
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: model-server
  name: model-server
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: model-server
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: model-server
    spec:
      containers:
      - args:
        - -t
        - "120"
        - -b
        - "0.0.0.0"
        - app:flask_app
        command:
        - gunicorn
        env:
        - name: ENV
          value: staging
        - name: GCP
          value: "2"
        image: gcr.io/my-production/my-model-server: myGitHash
        imagePullPolicy: Always
        name: model-server
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        resources:
          limits:
           nvidia.com/gpu: 1
        ports:
          - containerPort: 8000
            protocol: TCP
        volumeMounts:
        - name: model-files
          mountPath: /model-server/models
      # These containers are run during pod initialization
      initContainers:
      - name: model-download
        image: gcr.io/my-production/my-model-server: myGitHash
        command:
        - gsutil
        - cp
        - -r
        - gs://my-staging-models/*
        - /model-files/
        volumeMounts:
        - name: model-files
          mountPath: "/model-files"
      volumes:
      - name: model-files
        emptyDir: {}
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsUser: 0
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: model-server
  name: model-server-service
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8000
  selector:
    app: model-server
  sessionAffinity: None
  type: ClusterIP

我的问题开始了。我正在创建一组新任务。对于这组新任务,我将需要大量内存,因此我不想使用以前的服务。我想将其作为单独的新服务的一部分来进行。带有以下网址http://model-server-heavy-service.default.svc.cluster.local的内容。我试图创建一个新的Yaml文件model-server-heavy.yaml。在这个新的yaml文件中,我将服务的名称从model-server-service更改为model-server-heavy-service。另外,我将应用程序的名称和名称从model-server更改为model-sever-heavy。因此,最终的yaml文件看起来就像我在本文结尾处放置的一样。不幸的是,新模型服务器无法正常工作,我在kubernetes上收到有关新模型服务器的以下消息。

model-server-asdhjs-asd            1/1     Running            0          21m
model-server-heavy-xnshk   0/1     **CrashLoopBackOff**   8          21m

有人可以告诉我我在做错什么,以及我打算怎么做?为什么收到第二台模型服务器的消息 CrashLoopBackOff ?对于第二台模型服务器,我没有正确执行什么操作?

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: model-server-heavy
  name: model-server-heavy
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: model-server-heavy
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: model-server-heavy
    spec:
      containers:
      - args:
        - -t
        - "120"
        - -b
        - "0.0.0.0"
        - app:flask_app
        command:
        - gunicorn
        env:
        - name: ENV
          value: staging
        - name: GCP
          value: "2"
        image: gcr.io/my-production/my-model-server:mgGitHash
        imagePullPolicy: Always
        name: model-server-heavy
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        resources:
          limits:
           nvidia.com/gpu: 1
        ports:
          - containerPort: 8000
            protocol: TCP
        volumeMounts:
        - name: model-files
          mountPath: /model-server-heavy/models
      # These containers are run during pod initialization
      initContainers:
      - name: model-download
        image: gcr.io/my-production/my-model-server:myGitHash
        command:
        - gsutil
        - cp
        - -r
        - gs://my-staging-models/*
        - /model-files/
        volumeMounts:
        - name: model-files
          mountPath: "/model-files"
      volumes:
      - name: model-files
        emptyDir: {}
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsUser: 0
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: model-server-heavy
  name: model-server-heavy-service
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8000
  selector:
    app: model-server-heavy
  sessionAffinity: None
  type: ClusterIP

1 个答案:

答案 0 :(得分:1)

由于@ dawid-kruk和@ patrick-w,我必须在model-sever-heavy.yaml中进行两次修改才能使其正常工作。

  1. 将mountPath从/model-server-heavy/models更改为/model-server/models

  2. model-sever-heavy.yaml文件的第38行中,我应该将名称从model-server-heavy更改为model-sever

我首先尝试通过应用项目1来解决此问题,但没有成功。然后我也尝试了第二项并修复了。我需要同时安装1和2才能使服务器正常工作。我了解为什么我必须对第一项进行更改,但不确定第二项。