Question

我正在尝试将我的工作连接到一个容器中，该容器会将事件发送到另一个容器中的kafka集群。无论我尝试了什么，我都无法将活动发送给kafka主题我已经尝试了telnet和kafkacat到我的kafka的地址侦听器端口，一切正常：

这是我的工作撰写文件，“ 172.16.33.91”是我的本地IP地址：

version: '3'
services:
    events-processor:
        build:
            context: ./events-processor
        extra_hosts:
            - "host:172.16.33.91"
        restart: unless-stopped

这是我的工作代码，它将数据从1-> 1000发送到一个已有的主题 num-test ：

from time import sleep
from json import dumps
from kafka import KafkaProducer

if __name__=="__main__":
    producer = KafkaProducer(bootstrap_servers=['host:9093'],
                             value_serializer=lambda x: dumps(x).encode('utf-8'))
    for e in range(1000):
        data = {'number' : e}
        producer.send('numtest', value=data)
        print(data)
        sleep(5)

这是我的kafka-zookeeper撰写文件：

version: '3'

services: 
    zookeeper:
        image: confluentinc/cp-zookeeper
        environment:
            ZOOKEEPER_CLIENT_PORT: 2181
            ZOOKEEPER_TICK_TIME: 2000
        volumes:
            - zk-data:/var/lib/zookeeper/data
            - zk-logs:/var/lib/zookeeper/log
            - secrets:/etc/zookeeper/secrets
        restart: unless-stopped

    kafka:
        image: confluentinc/cp-kafka
        depends_on:
            - zookeeper
        ports: 
            - "9093:9093"
        environment:
            KAFKA_BROKER_ID: 1
            KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
            KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL://:9093
            KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT      
            KAFKA_LISTENERS: INTERNAL://:9092,EXTERNAL://:9093
            KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
            KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
        volumes:
            - kafka-data:/var/lib/kafka/data
            - secrets:/etc/kafka/secrets
        restart: unless-stopped

volumes:
    zk-logs: {}
    zk-data: {}
    kafka-data: {}
    secrets: {}

任何人都知道我做错了什么吗？任何帮助表示赞赏！

Answer 1

我记得在连接方面遇到类似的问题。事实证明，要使您的容器能够访问主机上的端口，您可能需要向防火墙添加其他规则以将其打开。

例如，对于iptables，您可以添加如下规则，以使主机可以接受来自docker容器的请求。

-A INPUT -i docker0 -j ACCEPT

或者，您可以将作业容器与kafka / zookeeper放在相同的docker网络上。通过将您的应用程序放置在相同的docker-compose文件中，或使用公共外部docker网络。

Answer 2

如果应该在更好的位置放置此信息，或者缺少寻找解决方案的关键字，请告知社区

已经有很多博客

KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,HOST://localhost:9093
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,HOST:PLAINTEXT      
KAFKA_LISTENERS: INTERNAL://0.0.0.0:9092,HOST://0.0.0.0:9093

使用://:port地址只会侦听本地主机名，是否侦听容器
您必须通过绑定到所有接口0.0.0.0
您必须在内部发布和/或任何外部主机名

Answer 3

这是另一个工作示例：

version: '2'
services:
  zookeeper1:
    image: confluentinc/cp-zookeeper:$CP_VERSION
    container_name: zookeeper1
    hostname: zookeeper1
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: "zookeeper1:22888:23888;zookeeper2:22888:23888;zookeeper3:22888:23888"
    ports:
      - 2181:2181
    volumes: 
      - ./data/zoo-1/data:/var/lib/zookeeper/data
      - ./data/zoo-1/log:/var/lib/zookeeper/log

  zookeeper2:
    image: confluentinc/cp-zookeeper:$CP_VERSION
    container_name: zookeeper2
    hostname: zookeeper2
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SERVER_ID: 2
      ZOOKEEPER_SERVERS: "zookeeper1:22888:23888;zookeeper2:22888:23888;zookeeper3:22888:23888"
    ports:
      - 2182:2181
    volumes: 
      - ./data/zoo-2/data:/var/lib/zookeeper/data
      - ./data/zoo-2/log:/var/lib/zookeeper/log

  zookeeper3:
    image: confluentinc/cp-zookeeper:$CP_VERSION
    container_name: zookeeper3
    hostname: zookeeper3
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SERVER_ID: 3
      ZOOKEEPER_SERVERS: "zookeeper1:22888:23888;zookeeper2:22888:23888;zookeeper3:22888:23888"
    ports:
      - 2183:2181
    volumes: 
      - ./data/zoo-3/data:/var/lib/zookeeper/data
      - ./data/zoo-3/log:/var/lib/zookeeper/log

  kafka1:
    image: confluentinc/cp-kafka:$CP_VERSION
    container_name: kafka1
    hostname: kafka1
    depends_on:
      - zookeeper1
      - zookeeper2
      - zookeeper3
    ports:
      - 9091:9091
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181,zookeeper2:2181,zookeeper3:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INTERNAL://:29092,OUTSIDE://:9091
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:29092,OUTSIDE://kafka1:9091
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/logs


  kafka2:
    image: confluentinc/cp-kafka:$CP_VERSION
    container_name: kafka2
    hostname: kafka2
    depends_on:
      - zookeeper1
      - zookeeper2
      - zookeeper3
    ports:
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181,zookeeper2:2181,zookeeper3:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INTERNAL://:29092,OUTSIDE://:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:29092,OUTSIDE://kafka2:9092
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/logs

  kafka3:
    image: confluentinc/cp-kafka:$CP_VERSION
    container_name: kafka3
    hostname: kafka3
    depends_on:
      - zookeeper1
      - zookeeper2
      - zookeeper3
    ports:
      - 9093:9093
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper1:2181,zookeeper2:2181,zookeeper3:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INTERNAL://:29092,OUTSIDE://:9093
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:29092,OUTSIDE://kafka3:9093
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_LOG_DIRS: /var/lib/kafka/logs

不要忘记编辑 /etc/hosts 和正确的主机名，以便能够在 Docker 之外使用上述 kafka 集群。

我的 /etc/hosts 文件如下所示：

127.0.0.1       localhost
127.0.0.1       kafka1
127.0.0.1       kafka2
127.0.0.1       kafka3

无法连接到网络外部的kafka

3 个答案: