Question

我目前正在尝试使用kafka和spring-kafka来消费消息。

但是我在为同一主题执行几个消费者时遇到了麻烦，并且有几个问题：

1 - 我的消费者在一段时间后会断开连接而无法重新连接

我的消费者会定期提出以下警告：

2017-09-06 15:32:35.054  INFO 5203 --- [nListener-0-C-1] f.b.poc.crawler.kafka.KafkaListener      : Consuming {"some-stuff": "yes"} from topic [job15]
2017-09-06 15:32:35.054  INFO 5203 --- [nListener-0-C-1] f.b.p.c.w.services.impl.CrawlingService  : Start of crawling
2017-09-06 15:32:35.054  INFO 5203 --- [nListener-0-C-1] f.b.p.c.w.services.impl.CrawlingService  : Url has already been treated ==> skipping
2017-09-06 15:32:35.054  WARN 5203 --- [nListener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : Auto-commit of offsets {job15-3=OffsetAndMetadata{offset=11547, metadata=''}, job15-2=OffsetAndMetadata{offset=15550, metadata=''}} failed for group group-3: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
2017-09-06 15:32:35.054  INFO 5203 --- [nListener-0-C-1] o.a.k.c.c.internals.ConsumerCoordinator  : Revoking previously assigned partitions [job15-3, job15-2] for group group-3
2017-09-06 15:32:35.054  INFO 5203 --- [nListener-0-C-1] s.k.l.ConcurrentMessageListenerContainer : partitions revoked:[job15-3, job15-2]
2017-09-06 15:32:35.054  INFO 5203 --- [nListener-0-C-1] o.a.k.c.c.internals.AbstractCoordinator  : (Re-)joining group group-3

这会导致消费者停下来等待几秒钟。

正如消息中所提到的，我将消费者session.timeout.ms增加到类似30000的内容。我仍然得到消息。正如您在提供的日志中看到的那样，在记录完成其过程后立即断开连接。所以......在30多岁的无生命之前很多。

2-两个消费者应用程序经常收到相同的消息

看着我的消费者＆＃39;日志我看到他们倾向于对待同一条消息。我理解卡夫卡是at-least-once，但我从没想过会遇到很多重复。希望我使用redis但我可能错过了一些我需要做的调整/属性。

代码

注意：我使用ConcurrentMessageListenerContainer与auto-commit=true但使用1个线程运行。我只是启动同一个应用程序的几个实例，因为消费者使用的是不是线程安全的服务。

KafkaContext.java

@Slf4j
@Configuration
@EnableConfigurationProperties(value = KafkaConfig.class)
class KafkaContext {

    @Bean(destroyMethod = "stop")
    public ConcurrentMessageListenerContainer kafkaInListener(IKafkaListener listener, KafkaConfig config) {
        final ContainerProperties containerProperties =
                new ContainerProperties(config.getIn().getTopic());
        containerProperties.setMessageListener(listener);
        final DefaultKafkaConsumerFactory<Integer, String> defaultKafkaConsumerFactory =
                new DefaultKafkaConsumerFactory<>(consumerConfigs(config));

        final ConcurrentMessageListenerContainer messageListenerContainer =
                new ConcurrentMessageListenerContainer<>(defaultKafkaConsumerFactory, containerProperties);

        messageListenerContainer.setConcurrency(config.getConcurrency());
        messageListenerContainer.setAutoStartup(false);
        return messageListenerContainer;
    }

    private Map<String, Object> consumerConfigs(KafkaConfig config) {
        final String kafkaHost = config.getHost() + ":" + config.getPort();
        log.info("Crawler_Worker connecting to kafka at {} with consumerGroup {}", kafkaHost, config.getIn().getGroupId());
        final Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaHost);
        props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, true);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, config.getIn().getGroupId());
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, JacksonNextSerializer.class);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, IntegerDeserializer.class);
        props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 30000);
        return props;
    }

}

监听

@Slf4j
@Component
class KafkaListener implements IKafkaListener {

    private final ICrawlingService crawlingService;

    @Autowired
    public KafkaListener(ICrawlingService crawlingService) {
        this.crawlingService = crawlingService;
    }

    @Override
    public void onMessage(ConsumerRecord<Integer, Next> consumerRecord) {
        log.info("Consuming {} from topic [{}]", JSONObject.wrap(consumerRecord.value()), consumerRecord.topic());

        consumerService.apply(consumerRecord.value());
    }
}

Answer 1

这里的主要问题是您的消费者群体不断被重新平衡。你是正确的增加session.timeout.ms，但我没有在你的配置中看到这个配置。尝试删除：

props.put(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG, 30000);

并设置：

props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 10);
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 30000);

您可以增加MAX_POLL_RECORDS_CONFIG以获得与经纪人沟通的更好表现。但是如果你只在一个线程中处理消息，那么保持这个值较低会更安全。

Spring-kafka和kafka 0.10

1 个答案: