Question

我有一个配置，用于使用Docker在Kubernetes上为Airflow部署4个Pod（因此，有4个工作程序）。但是，worker-0突然无法发出某些卷曲请求，而其他工人却可以发出卷曲请求。这导致管道发生故障。

我尝试阅读有关配置和有状态集不匹配的信息，但就我而言，所有工作人员都有一个配置，这是唯一的真理来源。

statefulsets-workers.yaml文件如下：

# Workers are not in deployment, but in StatefulSet, to allow each worker expose a mini-server
# that only serve logs, that will be used by the web server.

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: {{ template "airflow.fullname" . }}-worker
  labels:
    app: {{ template "airflow.name" . }}-worker
    chart: {{ template "airflow.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
spec:
  serviceName: "{{ template "airflow.fullname" . }}-worker"
  updateStrategy:
    type: RollingUpdate
  # Use experimental burst mode for faster StatefulSet scaling
  #   https://github.com/kubernetes/kubernetes/commit/****
  podManagementPolicy: Parallel
  replicas: {{ .Values.celery.num_workers }}
  template:
    metadata:
      {{- if .Values.airflow.pallet.config_path }}
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
      {{- end }}
      labels:
        app: {{ template "airflow.name" . }}-worker
        release: {{ .Release.Name }}
    spec:
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsUser: 1002
        fsGroup: 1002
      containers:
        - name: {{ .Chart.Name }}-worker
          imagePullPolicy: {{ .Values.airflow.image_pull_policy }}
          image: "{{ .Values.airflow.image }}:{{ .Values.airflow.imageTag }}"
          volumeMounts:
            {{- if .Values.airflow.storage.enabled }}
            - name: google-cloud-key
              mountPath: /var/secrets/google
              readOnly: true
            {{- end }}
            - name: worker-logs
              mountPath: /usr/local/airflow/logs
            - name: data
              mountPath: /usr/local/airflow/rootfs
          env:
            {{- if .Values.airflow.storage.enabled }}
            - name: GOOGLE_APPLICATION_CREDENTIALS
              value: /var/secrets/google/key.json
            {{- end }}
            {{- range $setting, $option := .Values.airflow.config }}
            - name: {{ $setting }}
              value: {{ $option }}
            {{- end }}
          securityContext:
            allowPrivilegeEscalation: false
          envFrom:
            - configMapRef:
                name: pallet-env-file
          args: ["worker"]
          ports:
            - name: wlog
              containerPort: 8793
              protocol: TCP
      {{- if .Values.airflow.image_pull_secret }}
      imagePullSecrets:
        - name: {{ .Values.airflow.image_pull_secret }}
      {{- end }}
      {{- if .Values.airflow.storage.enabled }}
      volumes:
        - name: google-cloud-key
          secret:
            secretName: {{ .Values.airflow.storage.secretName }}
      {{- end }}
  volumeClaimTemplates:
    - metadata:
        name: worker-logs
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 50Gi
    - metadata:
        name: data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 50Gi

我希望所有工作人员都能够连接到我向curl发出请求的服务。

Answer 1

事实证明，环境确实相同，但是接收机器没有将新IP地址列入白名单。

当所有Pod崩溃时，他们将节点带到一起，然后重新启动该节点，为其提供了新IP。因此，该节点上的工作程序的连接超时。

如何为所有气流中的工人（容器）确保相同的环境？

1 个答案: