Question

我们的Java应用程序中的Kafka使用者出于某种原因重新使用了该主题的消息。

生产者将消息发送到具有四个分区的主题。我们有一个消费者使用此主题的消息。应用程序在工作日例外周末一直运行：它在周末不会调用poll方法消费者配置：自动提交，自动提交时间为5秒（默认）。

应用程序运行正常，直到它再次恢复调用poll方法的星期日。我们从该主题中看到了数百万条消息。消费者基本上是从主题中查询所有消息。将新偏移量与周末停止之前的偏移量进行比较时。新偏移量要小得多，就像重置为四个分区中的所有分区一样。

我们不知道消费者方面发生了什么，因为它没有调用poll方法，因此没有打印日志消息。我们检查了kafka服务器日志，但我们什么也没找到。

以前有人看过这个吗？消费者会错误配置吗？

    <spring.kafka.version>1.1.2.RELEASE</spring.kafka.version>
     ... 
  <bean id="defaultKafkaConsumer"
        class="org.apache.kafka.clients.consumer.KafkaConsumer">
        <constructor-arg>  
            <map>
                <entry key="bootstrap.servers"  value="${kafka.bootstrap.servers}"></entry>
                <entry key="max.block.ms"  value="5000"></entry>
                <entry key="group.id"  value="kafkaconnect.tca"></entry>
                <entry key="auto.offset.reset" value="earliest"></entry>
                <entry key="key.deserializer"  value="org.apache.kafka.common.serialization.StringDeserializer"></entry>
                <entry key="value.deserializer"  value="org.apache.kafka.common.serialization.StringDeserializer"></entry>
            </map>  
        </constructor-arg>  
    </bean>

    getKafkaConsumer().subscribe(Arrays.asList(getKafkaTopic()));
            // set up the polling task
            handler = timer.scheduleAtFixedRate(new Runnable() {
                public void run() {
                    try {
                        processPoll();
                    } catch (Throwable t) {
                        LOG.error(String.format("error processing poll for inet: %s, details: %s - %s", getId(), t.getMessage(), t.getCause()), t);
                    }
                }
            }, 3, 3, TimeUnit.MILLISECONDS);



   processPoll() Method: destination will not be ready during the weekend.
   try {
            if (!isDestinationReady()) {
                if (destinationIgnoreCnt++ ==0) {
                    LOG.warn(String.format("outbound destination session is not ready - trying: %s/%s",destinationIgnoreCnt,destinationwaitingloop));
                } else if ((destinationIgnoreCnt++ % destinationwaitingloop) == 0) {
                    LOG.warn(String.format("outbound destination session is not ready - trying %s/%s", destinationIgnoreCnt,destinationwaitingloop));
                    destinationIgnoreCnt = 1;
                }
                messageIgnoreCnt = 0;
                return;
            }
            if(!isDestinationOpen()) {
                if (destinationIgnoreCnt++ ==0) {
                    LOG.error(String.format("outbound destination is not opended - trying:%s/%s.", destinationIgnoreCnt,destinationwaitingloop) );
                } else if ((destinationIgnoreCnt++ % destinationwaitingloop) == 0) {
                    LOG.error(String.format("outbound destination is not opended - trying %s/%s.", destinationIgnoreCnt,destinationwaitingloop));
                    destinationIgnoreCnt = 1;
                }
                messageIgnoreCnt = 0;
                return;
            }

            if (messageIgnoreCnt++ == 0) {
                LOG.info(String.format("kafka poller started. poll interval %s wait: %s", pollingInterval, 60000));
            } else if ((messageIgnoreCnt++ % 30) == 0) {// approximately 30mins
                LOG.info(String.format("kafka poller started. poll interval %s wait %s", pollingInterval, 60000));
                messageIgnoreCnt = 1;
            }

            if (getKafkaConsumer() == null) {
                LOG.critical("KafkaListener consumer is null");
                return;
            }

            ConsumerRecords<String, String> records = getKafkaConsumer().poll(60000);
            if (records == null || records.isEmpty()) {
                LOG.debug("zero records received from Kafka");
                return;
            }
            for (ConsumerRecord<String, String> record : records) {
                LOG.info(String.format("consuming from topic =  %s ", record.toString()));
                try {
                    String jsonMsg = record.value();

                    DirectBatchRequest payload = JacksonUtilities.getObjectMapper().readValue(jsonMsg, DirectBatchRequest.class);

                    if (payload != null) {
                        LOG.info(String.format("Got it reportId:%s", payload.getDestinationId()));
                        if(payload.getDestinationId() == 0) {
                            LOG.info(String.format("Applying default destination desk:%s", defaultDeskId));
                            payload.setDestinationId(defaultDeskId);
                        }
                        List<RequestEntryType> requestEntryTypeList = ((StreamDirectRequest) payload).getRequestList();
                        LOG.info(String.format("Processing size: %s" , requestEntryTypeList.size()) );
                        processRequest((StreamDirectRequest) payload);  //async call
                        LOG.info(String.format("Processing size: %s sent to Steam" , requestEntryTypeList.size()) );    
                    }
                } catch (Throwable t) {
                    LOG.error(String.format("KafkaListener JSON%s conversion error %s", record, t.getMessage()));
                }

            }

        } catch (Throwable t) {
            LOG.error(String.format("KafkaListener exception %s", t.getMessage()));

        }

Answer 1

如果消费者群体没有活动，Kafka默认会删除offsets.retention.minutes之后的偏移量。默认保留期为1440分钟（1天）。

在您的情况下，由于消费者群体在周末休息，因此重置偏移量。

请参阅https://kafka.apache.org/documentation/#brokerconfigs

Kafka消费者正在重构主题

1 个答案: