Question

我正在尝试从本地计算机上的外部kubernetes访问kafka。我正在使用spring应用程序来生成有关主题的事件。这是我的kafka部署文件：

kind: Deployment
metadata:
  name: kafka-broker0
  labels:
    app: kafka
spec:
  replicas: 2
  selector:
    matchLabels:
      app: kafka
      id: "0"
  template:
    metadata:
      labels:
        app: kafka
        id: "0"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "30718"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: 192.168.1.240
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181
        - name: KAFKA_BROKER_ID
          value: "0"
        - name: KAFKA_CREATE_TOPICS
          value: LaunchScraper:1:1

和服务文件ID：

kind: Service
metadata:
  name: kafka-services
  labels:
    name: kafka
spec:
  selector:
    app: kafka
    id: "0"
  ports:
    - protocol: TCP
      name: kafka-port
      port: 9092
  type: NodePort

我已经在kubernetes上创建了一个zookeeper吊舱。我的Spring Boot应用程序显示此错误：

2020-09-25 23:56:29.123  WARN 44324 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Connection to node -1 (/192.168.1.240:9092) could not be established. Broker may not be available.

Answer 1

似乎您尚未修复服务中的nodePort。达到您在KAFKA_ADVERTISED_PORT中输入的值时。还要将KAFKA_ADVERTISED_HOST设置为K8s节点的主机名/ DNS。

在您需要的规格中，在nodePort: 30718条目下添加ports。然后在您的客户端中，尝试使用节点的地址或主机名在30718端口上进行连接

此外，如果您打算在生产环境中部署Kafka，我建议您使用Strimzi https://Strimzi.io之类的运算符

Answer 2

在 Kubernetes 上部署 Kafka 实际上并不像我最初想象的那么简单，但经过多次试验和错误后，它确实奏效了。您在互联网上找到的许多示例对当前版本的 Kubernetes / Kafka 都不起作用。有效的是：

为 Kafka 使用 StatefulSet，而不是部署
如下设置KAFKA_ADVERTISED_LISTENERS和KAFKA_LISTENERS
NodePort 服务中用于外部访问的额外端口（在我的情况下为 32092，但它是任意的），然后不要忘记通过 32092 从外部访问 Kafka，而不是 9092

一个工作示例配置将是（作为您的部署和服务的替代品，可能不是最小的）：

apiVersion: v1
kind: Service
metadata:
  labels:
    service: kafka
  name: kafka
spec:
  type: NodePort
  ports:
  - name: "9092"
    port: 9092
    protocol: TCP
    targetPort: 9092
  - name: "9093"
    port: 9093
    protocol: TCP
    targetPort: 9093
  - name: "32092"
    port: 32092
    protocol: TCP
    targetPort: 32092
    nodePort: 32092
  selector:
    service: kafka-instance
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    service: kafka-instance
  name: kafka-instance
spec:
  selector:
    matchLabels:
      service: kafka-instance
  serviceName: "kafka"
  replicas: 1
  template:
    metadata:
      labels:
        service: kafka-instance
    spec:
      containers:
      - env:
        - name: MY_HOST_IP
          valueFrom:
            fieldRef:
              fieldPath: status.hostIP
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: KAFKA_ADVERTISED_LISTENERS
          value: INTERNAL://$(MY_POD_NAME).kafka.default.svc.cluster.local:9093,CLIENT://$(MY_POD_NAME).kafka.default.svc.cluster.local:9092,EXTERNAL://$(MY_HOST_IP):32092
        - name: KAFKA_INTER_BROKER_LISTENER_NAME
          value: INTERNAL
        - name: KAFKA_LISTENERS
          value: INTERNAL://:9093,CLIENT://:9092,EXTERNAL://:32092
        - name: KAFKA_LISTENER_SECURITY_PROTOCOL_MAP
          value: INTERNAL:PLAINTEXT,CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
        - name: KAFKA_PORT
          value: "9092"
        - name: KAFKA_RESTART_ATTEMPTS
          value: "10"
        - name: KAFKA_RESTART_DELAY
          value: "5"
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181
        - name: KAFKA_ZOOKEEPER_SESSION_TIMEOUT
          value: "6000"
        - name: ZOOKEEPER_AUTOPURGE_PURGE_INTERVAL
          value: "0"
        image: wurstmeister/kafka
        name: kafka-instance
        ports:
        - containerPort: 9092

如果您还没有zookeeper，只需添加它，它应该可以工作：

apiVersion: v1
kind: Service
metadata:
  labels:
    service: zoo1
  name: zoo1
spec:
  ports:
  - name: "2181"
    port: 2181
    targetPort: 2181
  selector:
    service: zoo1-instance
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    service: zoo1-instance
  name: zoo1-instance
spec:
  selector:
    matchLabels:
      service: zoo1-instance
  serviceName: "zoo1"
  replicas: 1
  template:
    metadata:
      labels:
        service: zoo1-instance
    spec:
      containers:
      - image: wurstmeister/zookeeper
        name: zoo1-instance
        ports:
        - containerPort: 2181

无法从外部kubernetes访问kafka

2 个答案: