Kafka消费者在Spring应用程序启动时进行重新平衡

时间：2018-12-15 14:30:20

标签： spring spring-boot concurrency apache-kafka kafka-consumer-api

我正在使用Spring Boot在Kafka上进行实验。

  • Spring Boot 2.1.0.RELEASE
  • Spring-Kafka 2.2.0

我有问题。如果有关于该主题的任何未读消息，则消费者平衡总是在启动我的应用程序时开始。

我有一个包含20个分区的主题，并且我有1个并发性为8的 KafkaListener ，并且我已将ThreadPoolTask​​Executor配置为：

@Bean
    ThreadPoolTaskExecutor messageProcessorExecutor() {
        ThreadPoolTaskExecutor exec = new ThreadPoolTaskExecutor();
        exec.setCorePoolSize(15);
        exec.setMaxPoolSize(25);
        exec.setKeepAliveSeconds(60);
        exec.setThreadNamePrefix("kafkaConsumer-");
        return exec;
    }

在应用程序启动时，通常只有kafkaConsumer-1（有时是kafkaConsumer-1和kafkaConsumer-2）才开始使用，对于其他7个使用者，我看到的日志如下所示； 

    2018-12-15 16:44:27.982  INFO 5551 --- [payment-group-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 16:44:32.050  INFO 5551 --- [payment-group-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 16:44:36.114  INFO 5551 --- [payment-group-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
    ....
        2018-12-15 16:44:57.841  INFO 5551 --- [kafkaConsumer-2] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] Group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
        2018-12-15 16:44:57.902  INFO 5551 --- [kafkaConsumer-3] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] Group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
        2018-12-15 16:44:57.956  INFO 5551 --- [kafkaConsumer-2] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] Discovered group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null)
        2018-12-15 16:44:57.956  INFO 5551 --- [kafkaConsumer-2] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] (Re-)joining group
        2018-12-15 16:44:57.999  INFO 5551 --- [kafkaConsumer-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] Group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
        2018-12-15 16:44:58.016  INFO 5551 --- [kafkaConsumer-3] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] Discovered group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null)
        2018-12-15 16:44:58.017  INFO 5551 --- [kafkaConsumer-3] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] (Re-)joining group
        2018-12-15 16:44:58.056  INFO 5551 --- [kafkaConsumer-5] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] Group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
        2018-12-15 16:44:58.064  INFO 5551 --- [kafkaConsumer-6] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] Group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
        2018-12-15 16:44:58.072  INFO 5551 --- [kafkaConsumer-7] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] Group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
        2018-12-15 16:44:58.085  INFO 5551 --- [kafkaConsumer-8] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] Group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
        2018-12-15 16:44:58.114  INFO 5551 --- [kafkaConsumer-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] Discovered group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null)
        2018-12-15 16:44:58.114  INFO 5551 --- [kafkaConsumer-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] (Re-)joining group
        2018-12-15 16:44:58.170  INFO 5551 --- [kafkaConsumer-5] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] Discovered group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null)
        2018-12-15 16:44:58.171  INFO 5551 --- [kafkaConsumer-5] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] (Re-)joining group
        2018-12-15 16:44:58.185  INFO 5551 --- [kafkaConsumer-6] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] Discovered group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null)
        2018-12-15 16:44:58.186  INFO 5551 --- [kafkaConsumer-6] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] (Re-)joining group
        2018-12-15 16:44:58.191  INFO 5551 --- [kafkaConsumer-7] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] Discovered group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null)
        2018-12-15 16:44:58.191  INFO 5551 --- [kafkaConsumer-7] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] (Re-)joining group
        2018-12-15 16:44:58.201  INFO 5551 --- [kafkaConsumer-8] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] Discovered group coordinator 127.0.0.1:9092 (id: 2147483646 rack: null)
        2018-12-15 16:44:58.202  INFO 5551 --- [kafkaConsumer-8] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] (Re-)joining group

在消耗一个线程的情况下，我看到以下内容：

    bash-4.4# $KAFKA_HOME/bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group payment-group-1 --describe
Warning: Consumer group 'payment-group-1' is rebalancing.

在完成对现有消息的使用之后，重新平衡也就完成了，我将看到类似以下的日志； 

    2018-12-15 16:46:12.415  INFO 5551 --- [kafkaConsumer-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Revoking previously assigned partitions [TopicY-4, TopicY-5, TopicY-6, TopicY-7, TopicY-8, TopicY-9, TopicY-10, TopicY-11, TopicY-0, TopicY-1, TopicY-2, TopicY-3, TopicY-12, TopicY-13, TopicY-14, TopicY-15, TopicY-16, TopicY-17, TopicY-18, TopicY-19]
2018-12-15 16:46:12.415  INFO 5551 --- [kafkaConsumer-1] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: [TopicY-4, TopicY-5, TopicY-6, TopicY-7, TopicY-8, TopicY-9, TopicY-10, TopicY-11, TopicY-0, TopicY-1, TopicY-2, TopicY-3, TopicY-12, TopicY-13, TopicY-14, TopicY-15, TopicY-16, TopicY-17, TopicY-18, TopicY-19]
2018-12-15 16:46:12.415  INFO 5551 --- [kafkaConsumer-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] (Re-)joining group
2018-12-15 16:46:12.482  INFO 5551 --- [kafkaConsumer-6] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] Successfully joined group with generation 25
2018-12-15 16:46:12.482  INFO 5551 --- [kafkaConsumer-3] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] Successfully joined group with generation 25
2018-12-15 16:46:12.482  INFO 5551 --- [kafkaConsumer-7] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] Successfully joined group with generation 25
2018-12-15 16:46:12.483  INFO 5551 --- [kafkaConsumer-7] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] Setting newly assigned partitions []
2018-12-15 16:46:12.483  INFO 5551 --- [kafkaConsumer-7] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: []
2018-12-15 16:46:12.484  INFO 5551 --- [kafkaConsumer-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] Successfully joined group with generation 25
2018-12-15 16:46:12.484  INFO 5551 --- [kafkaConsumer-4] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] Setting newly assigned partitions [TopicY-11]
2018-12-15 16:46:12.485  INFO 5551 --- [kafkaConsumer-8] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] Successfully joined group with generation 25
2018-12-15 16:46:12.485  INFO 5551 --- [kafkaConsumer-8] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] Setting newly assigned partitions []
2018-12-15 16:46:12.485  INFO 5551 --- [kafkaConsumer-8] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: []
2018-12-15 16:46:12.486  INFO 5551 --- [kafkaConsumer-2] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] Successfully joined group with generation 25
2018-12-15 16:46:12.486  INFO 5551 --- [kafkaConsumer-5] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] Successfully joined group with generation 25
2018-12-15 16:46:12.486  INFO 5551 --- [kafkaConsumer-5] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] Setting newly assigned partitions [TopicY-15]
2018-12-15 16:46:12.486  INFO 5551 --- [kafkaConsumer-2] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] Setting newly assigned partitions [TopicY-3]
2018-12-15 16:46:12.487  INFO 5551 --- [kafkaConsumer-3] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] Setting newly assigned partitions [TopicY-5]
2018-12-15 16:46:12.489  INFO 5551 --- [kafkaConsumer-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] Setting newly assigned partitions [TopicY-19]
2018-12-15 16:46:12.489  INFO 5551 --- [kafkaConsumer-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Successfully joined group with generation 25
2018-12-15 16:46:12.497  INFO 5551 --- [kafkaConsumer-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Setting newly assigned partitions [TopicY-0]
2018-12-15 16:46:12.509  INFO 5551 --- [kafkaConsumer-3] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-5]
2018-12-15 16:46:12.514  INFO 5551 --- [kafkaConsumer-6] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-19]
2018-12-15 16:46:12.517  INFO 5551 --- [kafkaConsumer-1] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-0]
2018-12-15 16:46:12.611  INFO 5551 --- [kafkaConsumer-2] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-3]
2018-12-15 16:46:12.612  INFO 5551 --- [kafkaConsumer-5] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-15]
2018-12-15 16:46:12.628  INFO 5551 --- [kafkaConsumer-4] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-11]
2018-12-15 16:47:44.563  INFO 5551 --- [kafkaConsumer-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 16:47:44.564  INFO 5551 --- [kafkaConsumer-4] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] Revoking previously assigned partitions [TopicY-11]
2018-12-15 16:47:44.564  INFO 5551 --- [kafkaConsumer-4] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: [TopicY-11]
2018-12-15 16:47:44.564  INFO 5551 --- [kafkaConsumer-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] (Re-)joining group
2018-12-15 16:47:44.577  INFO 5551 --- [kafkaConsumer-5] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 16:47:44.577  INFO 5551 --- [kafkaConsumer-2] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 16:47:44.577  INFO 5551 --- [kafkaConsumer-5] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] Revoking previously assigned partitions [TopicY-15]
2018-12-15 16:47:44.577  INFO 5551 --- [kafkaConsumer-2] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] Revoking previously assigned partitions [TopicY-3]
2018-12-15 16:47:44.577  INFO 5551 --- [kafkaConsumer-5] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: [TopicY-15]
2018-12-15 16:47:44.577  INFO 5551 --- [kafkaConsumer-2] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: [TopicY-3]
2018-12-15 16:47:44.577  INFO 5551 --- [kafkaConsumer-5] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] (Re-)joining group
2018-12-15 16:47:44.577  INFO 5551 --- [kafkaConsumer-2] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] (Re-)joining group
2018-12-15 16:47:44.577  INFO 5551 --- [kafkaConsumer-7] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 16:47:44.577  INFO 5551 --- [kafkaConsumer-7] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] Revoking previously assigned partitions []
2018-12-15 16:47:44.578  INFO 5551 --- [kafkaConsumer-7] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: []
2018-12-15 16:47:44.578  INFO 5551 --- [kafkaConsumer-7] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] (Re-)joining group
2018-12-15 16:47:44.586  INFO 5551 --- [kafkaConsumer-3] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 16:47:44.587  INFO 5551 --- [kafkaConsumer-6] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 16:47:44.587  INFO 5551 --- [kafkaConsumer-3] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] Revoking previously assigned partitions [TopicY-5]
2018-12-15 16:47:44.587  INFO 5551 --- [kafkaConsumer-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] Revoking previously assigned partitions [TopicY-19]
2018-12-15 16:47:44.587  INFO 5551 --- [kafkaConsumer-3] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: [TopicY-5]
2018-12-15 16:47:44.587  INFO 5551 --- [kafkaConsumer-6] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: [TopicY-19]
2018-12-15 16:47:44.587  INFO 5551 --- [kafkaConsumer-3] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] (Re-)joining group
2018-12-15 16:47:44.587  INFO 5551 --- [kafkaConsumer-6] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] (Re-)joining group
2018-12-15 16:47:44.589  INFO 5551 --- [kafkaConsumer-8] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 16:47:44.589  INFO 5551 --- [kafkaConsumer-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 16:47:44.589  INFO 5551 --- [kafkaConsumer-8] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] Revoking previously assigned partitions []
2018-12-15 16:47:44.589  INFO 5551 --- [kafkaConsumer-8] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: []
2018-12-15 16:47:44.589  INFO 5551 --- [kafkaConsumer-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Revoking previously assigned partitions [TopicY-0]
2018-12-15 16:47:44.589  INFO 5551 --- [kafkaConsumer-1] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: [TopicY-0]
2018-12-15 16:47:44.589  INFO 5551 --- [kafkaConsumer-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] (Re-)joining group
2018-12-15 16:47:44.589  INFO 5551 --- [kafkaConsumer-8] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] (Re-)joining group
2018-12-15 16:47:44.602  INFO 5551 --- [kafkaConsumer-3] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] Successfully joined group with generation 26
2018-12-15 16:47:44.602  INFO 5551 --- [kafkaConsumer-6] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] Successfully joined group with generation 26
2018-12-15 16:47:44.603  INFO 5551 --- [kafkaConsumer-3] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] Setting newly assigned partitions [TopicY-6, TopicY-7, TopicY-8]
2018-12-15 16:47:44.603  INFO 5551 --- [kafkaConsumer-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] Setting newly assigned partitions [TopicY-14, TopicY-15]
2018-12-15 16:47:44.603  INFO 5551 --- [kafkaConsumer-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Successfully joined group with generation 26
2018-12-15 16:47:44.604  INFO 5551 --- [kafkaConsumer-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Setting newly assigned partitions [TopicY-0, TopicY-1, TopicY-2]
2018-12-15 16:47:44.604  INFO 5551 --- [kafkaConsumer-7] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] Successfully joined group with generation 26
2018-12-15 16:47:44.605  INFO 5551 --- [kafkaConsumer-7] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] Setting newly assigned partitions [TopicY-16, TopicY-17]
2018-12-15 16:47:44.606  INFO 5551 --- [kafkaConsumer-2] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] Successfully joined group with generation 26
2018-12-15 16:47:44.606  INFO 5551 --- [kafkaConsumer-2] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] Setting newly assigned partitions [TopicY-4, TopicY-5, TopicY-3]
2018-12-15 16:47:44.610  INFO 5551 --- [kafkaConsumer-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] Successfully joined group with generation 26
2018-12-15 16:47:44.610  INFO 5551 --- [kafkaConsumer-5] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] Successfully joined group with generation 26
2018-12-15 16:47:44.611  INFO 5551 --- [kafkaConsumer-4] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] Setting newly assigned partitions [TopicY-9, TopicY-10, TopicY-11]
2018-12-15 16:47:44.611  INFO 5551 --- [kafkaConsumer-5] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] Setting newly assigned partitions [TopicY-12, TopicY-13]
2018-12-15 16:47:44.612  INFO 5551 --- [kafkaConsumer-8] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] Successfully joined group with generation 26
2018-12-15 16:47:44.612  INFO 5551 --- [kafkaConsumer-8] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] Setting newly assigned partitions [TopicY-18, TopicY-19]
2018-12-15 16:47:44.612  INFO 5551 --- [kafkaConsumer-6] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-14, TopicY-15]
2018-12-15 16:47:44.614  INFO 5551 --- [kafkaConsumer-3] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-6, TopicY-7, TopicY-8]
2018-12-15 16:47:44.618  INFO 5551 --- [kafkaConsumer-1] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-0, TopicY-1, TopicY-2]
2018-12-15 16:47:44.623  INFO 5551 --- [kafkaConsumer-7] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-16, TopicY-17]
2018-12-15 16:47:44.625  INFO 5551 --- [kafkaConsumer-2] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-4, TopicY-5, TopicY-3]
2018-12-15 16:47:44.626  INFO 5551 --- [kafkaConsumer-8] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-18, TopicY-19]
2018-12-15 16:47:44.628  INFO 5551 --- [kafkaConsumer-5] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-12, TopicY-13]
2018-12-15 16:47:44.877  INFO 5551 --- [kafkaConsumer-4] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-9, TopicY-10, TopicY-11]

我想在至少100 tps的应用程序中使用Kafka。但是，我对我提到的测试用例的实验感到困惑。重新平衡耗时太长，通常我只用一个线程消耗所有消息。我无法在生产中使用一个线程消耗所有消息。

注释：

  1. 我正在使用 @KafkaListener 注释编写消费者。
  2. 我尝试了许多消费者配置。

我当前的配置是：

ENABLE_AUTO_COMMIT_CONFIG=false
SESSION_TIMEOUT_MS_CONFIG=90000
HEARTBEAT_INTERVAL_MS_CONFIG=4000
AckMode=AckMode.RECORD
PollTimeout=10000

我正在寻求任何建议来解决我的问题。

顺便说一句，如果没有关于该主题的未读消息，则不会发生重新平衡，并且我会立即看到如下日志； 

    2018-12-15 17:12:01.618  INFO 5559 --- [kafkaConsumer-8] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] (Re-)joining group
2018-12-15 17:12:01.676  INFO 5559 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8081 (http) with context path ''
2018-12-15 17:12:01.684  INFO 5559 --- [           main] o.e.k.KafkaListenerApplication           : Started KafkaListenerApplication in 7.099 seconds (JVM running for 9.026)
2018-12-15 17:12:05.363  INFO 5559 --- [kafkaConsumer-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Attempt to heartbeat failed since group is rebalancing
2018-12-15 17:12:05.364  INFO 5559 --- [kafkaConsumer-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Revoking previously assigned partitions [TopicY-4, TopicY-5, TopicY-6, TopicY-7, TopicY-8, TopicY-9, TopicY-10, TopicY-11, TopicY-0, TopicY-1, TopicY-2, TopicY-3, TopicY-12, TopicY-13, TopicY-14, TopicY-15, TopicY-16, TopicY-17, TopicY-18, TopicY-19]
2018-12-15 17:12:05.364  INFO 5559 --- [kafkaConsumer-1] o.s.k.l.KafkaMessageListenerContainer    : partitions revoked: [TopicY-4, TopicY-5, TopicY-6, TopicY-7, TopicY-8, TopicY-9, TopicY-10, TopicY-11, TopicY-0, TopicY-1, TopicY-2, TopicY-3, TopicY-12, TopicY-13, TopicY-14, TopicY-15, TopicY-16, TopicY-17, TopicY-18, TopicY-19]
2018-12-15 17:12:05.364  INFO 5559 --- [kafkaConsumer-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] (Re-)joining group
2018-12-15 17:12:05.382  INFO 5559 --- [kafkaConsumer-5] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] Successfully joined group with generation 29
2018-12-15 17:12:05.382  INFO 5559 --- [kafkaConsumer-1] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Successfully joined group with generation 29
2018-12-15 17:12:05.382  INFO 5559 --- [kafkaConsumer-5] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-6, groupId=payment-group-1] Setting newly assigned partitions [TopicY-12, TopicY-13]
2018-12-15 17:12:05.384  INFO 5559 --- [kafkaConsumer-2] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] Successfully joined group with generation 29
2018-12-15 17:12:05.384  INFO 5559 --- [kafkaConsumer-7] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] Successfully joined group with generation 29
2018-12-15 17:12:05.384  INFO 5559 --- [kafkaConsumer-2] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-3, groupId=payment-group-1] Setting newly assigned partitions [TopicY-4, TopicY-5, TopicY-3]
2018-12-15 17:12:05.384  INFO 5559 --- [kafkaConsumer-7] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-8, groupId=payment-group-1] Setting newly assigned partitions [TopicY-16, TopicY-17]
2018-12-15 17:12:05.385  INFO 5559 --- [kafkaConsumer-4] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] Successfully joined group with generation 29
2018-12-15 17:12:05.385  INFO 5559 --- [kafkaConsumer-3] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] Successfully joined group with generation 29
2018-12-15 17:12:05.385  INFO 5559 --- [kafkaConsumer-4] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-5, groupId=payment-group-1] Setting newly assigned partitions [TopicY-9, TopicY-10, TopicY-11]
2018-12-15 17:12:05.385  INFO 5559 --- [kafkaConsumer-8] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] Successfully joined group with generation 29
2018-12-15 17:12:05.385  INFO 5559 --- [kafkaConsumer-3] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-4, groupId=payment-group-1] Setting newly assigned partitions [TopicY-6, TopicY-7, TopicY-8]
2018-12-15 17:12:05.385  INFO 5559 --- [kafkaConsumer-8] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-9, groupId=payment-group-1] Setting newly assigned partitions [TopicY-18, TopicY-19]
2018-12-15 17:12:05.385  INFO 5559 --- [kafkaConsumer-6] o.a.k.c.c.internals.AbstractCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] Successfully joined group with generation 29
2018-12-15 17:12:05.386  INFO 5559 --- [kafkaConsumer-6] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-7, groupId=payment-group-1] Setting newly assigned partitions [TopicY-14, TopicY-15]
2018-12-15 17:12:05.389  INFO 5559 --- [kafkaConsumer-1] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-2, groupId=payment-group-1] Setting newly assigned partitions [TopicY-0, TopicY-1, TopicY-2]
2018-12-15 17:12:05.400  INFO 5559 --- [kafkaConsumer-4] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-9, TopicY-10, TopicY-11]
2018-12-15 17:12:05.404  INFO 5559 --- [kafkaConsumer-8] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-18, TopicY-19]
2018-12-15 17:12:05.404  INFO 5559 --- [kafkaConsumer-6] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-14, TopicY-15]
2018-12-15 17:12:05.410  INFO 5559 --- [kafkaConsumer-1] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-0, TopicY-1, TopicY-2]
2018-12-15 17:12:05.505  INFO 5559 --- [kafkaConsumer-7] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-16, TopicY-17]
2018-12-15 17:12:05.510  INFO 5559 --- [kafkaConsumer-3] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-6, TopicY-7, TopicY-8]
2018-12-15 17:12:05.516  INFO 5559 --- [kafkaConsumer-2] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-4, TopicY-5, TopicY-3]
2018-12-15 17:12:05.520  INFO 5559 --- [kafkaConsumer-5] o.s.k.l.KafkaMessageListenerContainer    : partitions assigned: [TopicY-12, TopicY-13]

0 个答案:

没有答案
相关问题
最新问题