Question

我正在尝试编写一个简单的java kafka使用者来使用与https://github.com/bkimminich/apache-kafka-book-examples/blob/master/src/test/kafka/consumer/SimpleHLConsumer.java中类似的代码来读取数据。

看起来我的应用程序能够连接，但它没有获取任何数据。请建议。

import kafka.consumer.Consumer;
import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;

import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
//import scala.util.parsing.json.JSONObject
import scala.util.parsing.json.JSONObject;

public class SimpleHLConsumer {

    private final ConsumerConnector consumer;
    private final String topic;

    public SimpleHLConsumer(String zookeeper, String groupId, String topic) {
        Properties props = new Properties();
        props.put("zookeeper.connect", zookeeper);
        props.put("group.id", groupId);
      //  props.put("zookeeper.session.timeout.ms", "5000");
      //  props.put("zookeeper.sync.time.ms", "250");
      //  props.put("auto.commit.interval.ms", "1000");

        consumer = Consumer.createJavaConsumerConnector(new ConsumerConfig(props));
        this.topic = topic;
    }

    public void testConsumer() {
        Map<String, Integer> topicCount = new HashMap<>();
        topicCount.put(topic, 1);

        Map<String, List<KafkaStream<byte[], byte[]>>> consumerStreams = consumer.createMessageStreams(topicCount);
        System.out.println(consumerStreams);
        List<KafkaStream<byte[], byte[]>> streams = consumerStreams.get(topic);
        System.out.println(streams);
        System.out.println(consumer);
        for (final KafkaStream stream : streams) {
            ConsumerIterator<byte[], byte[]> it = stream.iterator();
            System.out.println("for loop");
            System.out.println(it);
            System.out.println("Message from Single Topic: " + new String(it.next().message()));
            //System.out.println("Message from Single Topic: " + new String(it.message()));
            while (it.hasNext()) {
                System.out.println("in While");
                System.out.println("Message from Single Topic: " + new String(it.next().message()));
            }
        }
       // if (consumer != null) {
       //     consumer.shutdown();
       // }
    }

    public static void main(String[] args) {
        String topic = "test";
        SimpleHLConsumer simpleHLConsumer = new SimpleHLConsumer("localhost:2181", "testgroup", topic);
        simpleHLConsumer.testConsumer();
    }

}

这是我在eclipse中看到的输出。它确实似乎连接到我的zookeeper，但它只是挂在那里，它根本不显示任何消息。

log4j:WARN No appenders could be found for logger (kafka.utils.VerifiableProperties).
log4j:WARN Please initialize the log4j system properly.
SLF4J: The requested version 1.6 by your slf4j binding is not compatible with [1.5.5, 1.5.6]
SLF4J: See http://www.slf4j.org/codes.html#version_mismatch for further details.
{test=[testgroup kafka stream]}
[testgroup kafka stream]
kafka.javaapi.consumer.ZookeeperConsumerConnector@6200f9cb
for loop

Answer 1

消费者迭代器hasNext阻止调用。如果没有可用于消费的新消息，它将无限期地阻止。

要验证这一点，请将代码更改为

// Comment 2 lines below
// System.out.println(it);
// System.out.println("Message from Single Topic: " + new String(it.next().message()));
// Line below is blocking. Your code will hang till next message in topic. 
// Add new message in topic using producer, message will appear in console
 while (it.hasNext()) {

更好的方法是在单独的线程中执行代码。使用consumer.timeout.ms指定以ms为单位的时间，之后使用者将抛出超时异常

// keepRunningThread is flag to control when to exit consumer loop
while(keepRunningThread) 
{
  try  
  {
    if(it.hasNext())
    {
        System.out.println(new String(it.next().message()));
    }
  } 
  catch(ConsumerTimeoutException ex) 
  {
    // Timeout exception waiting for kafka message
    // Wait for 5 (or t) seconds before checking for message again
     Thread.sleep(5000);
  }
}‍‍‍‍‍‍‍‍‍‍

kafka java消费者没有阅读数据

1 个答案: