Question

以下是kafka使用者的配置和代码。它在初始化时可以正常工作，但是当使用者在失败后重新启动时，它不会处理任何其他记录。 ENABLE_AUTO_COMMIT为false，SESSION_TIMEOUT为30秒，POLL_TIMEOUT为0秒

val props = new Properties
  props.put(BOOTSTRAP_SERVERS, config.getString(BOOTSTRAP_SERVERS).get)
  props.put(KEY_DESERIALIZATION, config.getString(KEY_DESERIALIZATION).get)
  props.put(VALUE_DESERIALIZATION, config.getString(VALUE_DESERIALIZATION).get)
  props.put(ENABLE_AUTO_COMMIT, config.getString(ENABLE_AUTO_COMMIT).get)
  props.put(GROUP_ID, "ng-evisit")
  props.put(SESSION_TIMEOUT, config.getNumber(SESSION_TIMEOUT).get)

  val consumer = new KafkaConsumer[String, String](props: Properties)
  val partitionInfo: List[PartitionInfo] =
    consumer.partitionsFor(config.getString(TOPIC).get).asScala.toList
  val partitions: util.Collection[TopicPartition] = partitionInfo
    .map(x => new TopicPartition(config.getString(TOPIC).get, x.partition()))
    .asJavaCollection

  consumer.assign(partitions)

while (true){
    try {
          val records = consumer.poll(config.getLong(POLL_TIMEOUT).get)
          for (record <- records.asScala) {
            val value = record.value
            logger.info(
              "Received messages from kafka: " + record.value)
            f(value)
            doCommitSync()
          }

        } catch {
          case e: WakeupException =>
            logger.error("Kafka Consumer Error: " + e.getMessage)
          case e: Throwable =>
            logger.error("Kafka Consumer Error: " + e.getMessage)
            throw e
        }
}

其中f是对收到的值执行的功能

def doCommitSync(): Unit = {
        try {
          consumer.commitSync()
        } catch {
          case e: WakeupException =>
            doCommitSync()
            throw e
          case e: CommitFailedException =>
            logger.error(e.getMessage)
        }
      }
      }

请建议使用首次使用kafka的消费者。很长时间以来一直在尝试解决此问题。预先感谢。

Kafka使用者重启后不使用消息

0 个答案: