Kakfa Consumer.poll(1000)首次超过4-5分钟

时间:2019-07-16 14:31:56

标签: apache-kafka

当我第一次点击consumer.poll(1000)时,我发现要花4-5分钟以上的时间才能返回消费者记录。有人知道为什么会这样吗?

此外,请注意,我已将auto-offset-reset-Config属性设置为latest,其中max-no-records属性设置为1000

这确实引起了主要问题,因为我的应用程序是多线程的。请任何建议都会有所帮助

Consumer<String, String> consumer = ConsumerCreator.createConsumer();
    int totalPollTimeMs = 0;
    int pollTimems = 1000;
    while (totalPollTimeMs < 5000) {
        ConsumerRecords<String, String> consumerRecords = consumer.poll(pollTimems);
        totalPollTimeMs += pollTimems;
        if (consumerRecords.count() > 0) {
            for (ConsumerRecord<String, String> consumerRecord : consumerRecords) {
                if (null != consumerRecord.value()) {
                    try {
                        ObjectMapper objectMapper = new ObjectMapper();
                        objectMapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false);
                        OutBoundRequest outBoundReq = objectMapper.readValue(consumerRecord.value(),OutBoundRequest.class);
                        ourBoundReqList.add(outBoundReq);
                    } catch (JsonParseException e) {

                    } catch (JsonMappingException e) {

                    } catch (IOException e) {

                    }
                }
            }
        }
    }

    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBrokerParam.getValue());
    props.put(ConsumerConfig.GROUP_ID_CONFIG, kafkaGroupIdparam.getValue());
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 1000);
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
    props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");
    Consumer<String, String> consumer = new KafkaConsumer<String, String>(props);
    consumer.subscribe(Arrays.asList(kafkaTopicparam.getValue()));

0 个答案:

没有答案