Question

我有卡夫卡高级消费者。

public class KafkaHighLevelConsumer implements Runnable {
    private final KafkaConsumer<String, String> consumer;
    private final List<String> topics;
    private final int id;

    public KafkaHighLevelConsumer(int id,
                        String groupId,
                        List<String> topics,BlockingQueue<String> storyQueue) {
        this.id = id;
        this.topics = topics;
        Properties props = new Properties();
        props.put("bootstrap.servers", "localhost:9091");
        props.put("group.id", groupId);
        props.put("key.deserializer", StringDeserializer.class.getName());
        props.put("value.deserializer", StringDeserializer.class.getName());
        this.consumer = new KafkaConsumer<>(props);
    }

    @Override
    public void run() {
        try {
            consumer.subscribe(topics);

            while (true) {
                ConsumerRecords<String, String> records = consumer.poll(100);
                for (ConsumerRecord<String, String> record : records) {
                    Map<String, Object> data = new HashMap<>();
                    data.put("partition", record.partition());
                    data.put("offset", record.offset());
                    data.put("value", record.value());
                    System.out.println(this.id + ": " + data);
                }
            }
        } catch (WakeupException e) {
            // ignore for shutdown
        }finally {
            consumer.close();
        }
    }

    public void shutdown() {
        consumer.wakeup();
    }
}

消费者工作正常，但我需要监控消费者的状态。如果服务器IP或端口不正确，为什么我们没有Exception？

如果我将端口更改为某个不正确的props.put("bootstrap.servers", "localhost:9091");到props.put("bootstrap.servers", "localhost:100500");，我仍然无法获得任何例外。

我想知道我是否成功连接到卡夫卡！有可能处理这种情况吗？

我使用这样的Maven deps

<dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka-clients</artifactId>
            <version>0.9.0.1</version>
        </dependency>

谢谢！

Answer 1

根据客户文档（https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html）：

它将透明地处理Kafka集群中服务器的故障

库无法区分临时故障，永久性故障或配置错误。所以它认为每次失败都是可以重试的，而且确实如此，永远重试，永远不会返回错误。

这是检查连接状态的一种解决方法：询问一些保证返回的信息，等待一段合理的时间，如果没有任何返回，那么你知道连接有问题：

int CONNECTION_TEST_TIMEOUT_SECONDS = 10; // or whatever is appropriate for your environment

ExecutorService executor = Executors.newSingleThreadExecutor();
Runnable testTask = consumer::listTopics;

Future future = executor.submit(testTask);
try {
    future.get(CONNECTION_TEST_TIMEOUT_SECONDS, TimeUnit.SECONDS);
} catch (TimeoutException te) {
    consumer.wakeup();
    throw new IOException("Could not communicate with the server within " + CONNECTION_TEST_TIMEOUT_SECONDS + " seconds");
} catch (InterruptedException e) {
    // Nothing to do. Maybe a warning in the log?
} catch (ExecutionException e) {
    throw new IOException("Exception while running connection test: " + e.getMessage(), e);
}

Answer 2

融合的Kafka OnError中有一个事件，只要使用者连接失败，就会触发错误。这是代码说明

    //
    // Summary:
    //     Raised on critical errors, e.g. connection failures or all brokers down. Note
    //     that the client will try to automatically recover from errors - these errors
    //     should be seen as informational rather than catastrophic
    //
    // Remarks:
    //     Executes on the same thread as every other Consumer event handler (except OnLog
    //     which may be called from an arbitrary thread).
    public event EventHandler<Error> OnError;

Assembly Confluent.Kafka，版本= 0.11.6.0，文化=中性， PublicKeyToken =空

如何检查卡夫卡的消费状况

2 个答案: