Question

I'm trying to send messages from Logstash to Kafka and keep getting the following error:

tail -f /var/log/logstash/logstash-plain.log
[2018-04-18T17:55:33,836][WARN ][org.apache.kafka.clients.NetworkClient] 
[Producer clientId=producer-1] Connection to node -1 could not be established. Broker may not be available.
[2018-04-18T17:55:46,095][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.1}
[2018-04-18T17:55:51,979][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.1}
[2018-04-18T17:56:46,196][INFO ][logstash.outputs.kafka   ] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.1}]

This was working previously, but not now and I'm not sure what changed since the machine was untouched.

I can communicate to Kafka through the console producer on the machine Kafka is running on, as well as the machine Logstash is running on, and from a remote machine out of the VPC, so I don't believe this is a network error. To be sure, I put Logstash on the same machine as Kafka, but I get the same error.

I double checked and made sure that I'm at the latest version of Kafka, Logstash, and the logstash-output-kafka plugin, so I don't think it's a compatibility issue either. I have tried restarting Logstash and the broker to no avail.

Basically, I'm at a loss at how to solve this issue. Any help would be appreciated. Thanks.

Answer 1

好吧，我明白了。这是logstash.conf文件中的配置问题：当我将kafka输出更改为计算机的本地IP地址时，它可以正常工作。它看起来并不适用于公共或私人主机名或公共IP地址。因为我正在使用带有多个kafka主题的条件，所以：

output {
  kafka {
    bootstrap_servers => "10.0.0.XX:9092"
    codec => json
    topic_id => "test_topic"
  }
  if [type] == "ping" {
    kafka {
      bootstrap_servers => "10.0.0.XX:9092"
      codec => json
      topic_id => "logstash-ping"
    }
  }
  else if [type] == "snmp" {
    kafka {
      bootstrap_servers => "10.0.0.XX:9092"
      codec => json
      topic_id => "logstash-snmp"
    }
  }
}

如果没有正确设置其中任何一个，它将无法发送给其中任何一个，尽管从我在logstash日志中看到的内容正确连接。

作为参考，我对listeners和advertised.listeners的设置是这样的：

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://ec2-XX-XX-XX-XX.compute-1.amazonaws.com:9092

Logstash plugin to output to Kafka not working

1 个答案: