Question

我在Kubernetes上安装了一个Kafka，其中两个代理在两个服务上运行。

NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
kafka-service    NodePort    10.101.47.36     <none>        9092:32132/TCP               2h
kafka-service2   NodePort    10.110.151.216   <none>        9092:32133/TCP               2h
kubernetes       ClusterIP   10.96.0.1        <none>        443/TCP                      1d
zoo1             ClusterIP   10.105.248.153   <none>        2181/TCP,2888/TCP,3888/TCP   2h

这两个都是分别暴露在32132和32133端口上的节点端口。

我按照以下规范创建了集群：

---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-broker1
spec:
  template:
    metadata:
      labels:
        app: kafka
        id: "1"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "32133"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: localhost
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181
        - name: KAFKA_BROKER_ID
          value: "1"
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: kafka-broker2
spec:
  template:
    metadata:
      labels:
        app: kafka
        id: "1"
    spec:
      containers:
      - name: kafka
        image: wurstmeister/kafka
        ports:
        - containerPort: 9092
        env:
        - name: KAFKA_ADVERTISED_PORT
          value: "32132"
        - name: KAFKA_ADVERTISED_HOST_NAME
          value: localhost
        - name: KAFKA_ZOOKEEPER_CONNECT
          value: zoo1:2181

这旋转了两个豆荚kafka-broker1和kafka-broker2。我为运行的Zookeeper提供了一项单独的服务，名为zoo1。

我尝试使用以下命令创建新主题：

bin/kafka-topics.sh --create --zookeeper 10.105.248.153:2181 --replication-factor 2 --partitions 2 --topic test

它被创建了。以下是对主题test

的描述

Topic:test  PartitionCount:2    ReplicationFactor:2 Configs:
    Topic: test Partition: 0    Leader: 2   Replicas: 2,1   Isr: 2
    Topic: test Partition: 1    Leader: 2   Replicas: 1,2   Isr: 2

但是当我尝试通过生产者发布有关此主题的消息时

kafka/bin/kafka-console-producer.sh --broker-list 10.110.151.216:9092,10.101.47.36:9092 --topic test 要么 kafka/bin/kafka-console-producer.sh --broker-list localhost:32132,localhost:32133 --topic test

我不断收到以下警告：

[2018-08-08 09:53:31,629] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:31,731] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 3 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:31,840] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:31,949] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 5 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:32,056] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:32,163] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 7 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:32,272] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 8 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:32,382] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 9 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2018-08-08 09:53:32,488] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 10 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

我不知道为什么会这样。

Answer 1

Kafka使用Zookeeper进行Kafka Broker和Topic Partition对的领导选举。 ZooKeeper返回POD的地址。如果重新创建POD，ZooKeeper将返回错误的地址。因此，最佳做法是不要使用POD的IP地址。

您使用的是wurstmeister/kafka图片，我在Readme中发现了有关KAFKA_ADVERTISED_HOST_NAME的一则注释：

如果要运行多个代理，请不要使用localhost或127.0.0.1作为主机ip。

您的安装中有2个代理，每个代理以localhost作为主机名。我认为这可能是您遇到问题的原因。尝试使用kafka1和kafka2代替localhost，并为/etc/hosts文件添加该名称的别名：

kafka1 127.0.0.1 kafka2 127.0.0.1

然后，您可以尝试连接到像这样的经纪人：

kafka/bin/kafka-console-producer.sh --broker-list kafka1:32132,kafka2:32133 --topic test

此外，第二次部署中似乎缺少KAFKA_BROKER_ID设置，但这可能是复制/粘贴问题。

Answer 2

确保Kafka外部端口和k8s服务nodePort一致，其他服务调用k8s-service：nodeport。写这个config_kafka_in_kubernetes。我回答了这个问题kafka-behind-traefik，希望对您有所帮助！

在Kubernetes上部署Kafka时出错

2 个答案: