卡夫卡连接问题

时间：2018-10-21 07:50:22

标签： ruby-on-rails amazon-web-services amazon-ec2 apache-kafka

我已经在docker swarm上安装了Kafka，但是我一直在面临以下问题。

谁能阐明什么是问题？而我在这里想念的是什么？

fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    | bundler: failed to load command: bin/subscriber_kafka (bin/subscriber_kafka)
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | I, [2018-10-21T07:37:41.204851 #1]  INFO -- : Leaving group `accounts_service.account`
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.204879 #1] DEBUG -- : Getting group coordinator for `accounts_service.account`
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/cluster.rb:121:in `get_group_coordinator'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.204927 #1] DEBUG -- : Opening connection to ip-10-0-1-109.ap-southeast-1.compute.internal:9094 with client id ruby-kafka...
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:41.204555 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:190:in `coordinator'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.206519 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:41.206599 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:131:in `block in join_group'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | I, [2018-10-21T07:37:41.206633 #1]  INFO -- : Leaving group `accounts_service.customer`
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.206660 #1] DEBUG -- : Getting group coordinator for `accounts_service.customer`
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.206717 #1] DEBUG -- : Opening connection to ip-10-0-1-109.ap-southeast-1.compute.internal:9094 with client id ruby-kafka...
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.222250 #1] DEBUG -- : Handling fetcher command: stop
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:21:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.222428 #1] DEBUG -- : Handling fetcher command: stop
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:35:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:130:in `join_group'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:41.222955 #1] DEBUG -- : Handling fetcher command: stop
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208336 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer_group.rb:47:in `join'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:46.208412 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208545 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:443:in `join_group'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208265 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:506:in `fetch_batches'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:46.208815 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.208867 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:206:in `block in each_message'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:394:in `block in consumer_loop'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.209073 #1] ERROR -- : Failed to connect to ip-10-0-1-109.ap-southeast-1.compute.internal:9094: getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | D, [2018-10-21T07:37:46.209110 #1] DEBUG -- : Closing socket to ip-10-0-1-109.ap-southeast-1.compute.internal:9094
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:21:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | E, [2018-10-21T07:37:46.209170 #1] ERROR -- : Failed to get coordinator info from ip-10-0-1-109.ap-southeast-1.compute.internal:9094 (node_id=109): getaddrinfo: Try again
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/instrumenter.rb:35:in `instrument'
fastlane_accounts_service_subscriber_kafka.1.z5uwnusziv2g@ip-10-0-1-109.ap-southeast-1.compute.internal    | bundler: failed to load command: bin/subscriber_kafka (bin/subscriber_kafka)
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:393:in `consumer_loop'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/ruby-kafka-0.7.2/lib/kafka/consumer.rb:205:in `each_message'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/kafka_tools-0.2.0/lib/kafka_tools/consumer.rb:65:in `listen_messages'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/kafka_tools-0.2.0/lib/kafka_tools/consumer.rb:20:in `run'
fastlane_accounts_service_subscriber_kafka.1.yoh6ytisqt3x@ip-10-0-1-109.ap-southeast-1.compute.internal    |   /app/vendor/bundle/ruby/2.5.0/gems/kafka_tools-0.2.0/lib/kafka_tools.rb:26:in `block (2 levels) in consume'

我的Docker Swarm配置

services:
  zookeeper:
    image: wurstmeister/zookeeper:latest
    networks:
      - fastlane
    deploy:
      mode: global

  kafka:
    image: wurstmeister/kafka:latest
    ports:
      - target: 9094
        published: 9094
        protocol: tcp
        mode: host
    environment:
      HOSTNAME_COMMAND: ${KAFKA_HOSTNAME_COMMAND}
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: INSIDE://:9092,OUTSIDE://_{HOSTNAME_COMMAND}:9094
      KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      BROKER_ID_COMMAND: ${BROKER_ID_COMMAND}
    networks:
      - fastlane
    depends_on:
      - zookeeper

  kafka_manager:
    image: deric/kafka-manager:1.3.3.18
    networks:
      - fastlane
    environment:
      ZK_HOSTS: zookeeper:2181
      APPLICATION_SECRET: letmein
    deploy:
      labels:
        - traefik.backend.loadbalancer.swarm=true
        - traefik.backend=kafka-manager
        - traefik.frontend.rule=Host:kafka-manager.${DOMAIN}
        - traefik.port=9000
        - traefik.docker.network=fastlane
    depends_on:
      - zookeeper

KAFKA_HOSTNAME_COMMAND = curl http://169.254.169.254/latest/meta-data/local-hostname
BROKER_ID_COMMAND="curl http://169.254.169.254/latest/meta-data/local-ipv4 | awk -F '.' '{print \$4}'"

1 个答案:

答案 0 :(得分：1)

您看过自述文件吗？它专门提到了在AWS中运行需要做的事情

  

如果指定了KAFKA_ADVERTISED_HOST_NAME，则优先于HOSTNAME_COMMAND

     

对于AWS部署，您可以使用元数据服务获取容器主机的IP：

     

HOSTNAME_COMMAND=wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/local-ipv4

     

参考：http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html

因此，首先，请先尝试一下文档所说的内容，然后删除KAFKA_HOSTNAME_COMMAND =的等号周围的空格

如果您在本地而不是在AWS网络中运行Ruby代码，则需要在AWS控制台的VPC设置中打开Kafka端口后，获取上述命令以返回外部地址。

我想提一下，如果您的Ruby应用程序（a）也是Compose文件的一部分，则连接到Kafka实例会更容易。例如，使用覆盖网络或Træfik，您可以直接访问kafka:9094服务。

相关问题
最新问题