Question

我想测试一个Kafka示例。我正在使用Kafka 0.10.0.1 制片人：

object ProducerApp extends App {

val topic = "topicTest"
val  props = new Properties()
props.put("bootstrap.servers", "localhost:9092")
props.put(ConsumerConfig.GROUP_ID_CONFIG, "consumer")
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer")
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer")
val producer = new KafkaProducer[String, String](props)
for(i <- 0 to 20)
{    
val record = new ProducerRecord(topic, "key "+i," value "+i)    
producer.send(record)    
Thread.sleep(100)    
}
}

消费者（使用1个分区创建主题“topicTest”）：

object ConsumerApp extends App {
val topic = "topicTest"  
val properties = new Properties
properties.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092")
properties.put(ConsumerConfig.GROUP_ID_CONFIG, "consumer")  
properties.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false")  
properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
properties.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
properties.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
val consumer = new KafkaConsumer[String, String](properties)  
consumer.subscribe(scala.List(topic).asJava)    
while (true) {
consumer.seekToBeginning(consumer.assignment())
val records:ConsumerRecords[String,String] = consumer.poll(20000)
println("records size "+records.count())
records.asScala.foreach(rec => println("offset "+rec.offset()))    
}  
}

问题在于消费者在第一次迭代时没有从偏移量0读取，而是在其他情况下读取。我想知道原因以及如何让消费者在所有迭代中从偏移0读取。预期结果是：

records size 6
offset 0
offset 1
offset 2
offset 3
offset 4
offset 5
records size 6
offset 0
offset 1
offset 2
offset 3
offset 4
offset 5
...

但获得的结果是：

records size 4
offset 2
offset 3
offset 4
offset 5
records size 6
offset 0
offset 1
offset 2
offset 3
offset 4
offset 5
...

Answer 1

我无法弄清楚究竟是什么错误，我写的代码与你的相同。但对我来说它工作正常。如果你想要，你可以使用下面的代码片段。

import java.util
import org.apache.kafka.clients.consumer.ConsumerRebalanceListener;
import org.apache.kafka.clients.consumer.KafkaConsumer
import org.apache.kafka.common.TopicPartition
import org.apache.kafka.common.serialization.LongDeserializer;
import scala.collection.JavaConverters._
import java.util.Properties

object ConsumerExample extends App {

  val TOPIC = "test-stack"

  val props = new Properties()

  props.put("bootstrap.servers", "localhost:9092")


  props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
  props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
  props.put("group.id", "testinf")
  props.put("auto.offset.reset", "earliest")
  props.put("auto.offset.reset.config", "false")

  var listener = new ConsumerRebalanceListener() {
    override def onPartitionsAssigned(partitions: util.Collection[TopicPartition]): Unit = {
      println("Assignment : " + partitions)

    }
    override def onPartitionsRevoked(partitions: util.Collection[TopicPartition]): Unit = {
      // do nothing
    }

  }

  val consumer = new KafkaConsumer[String, String](props)

  consumer.subscribe(util.Collections.singletonList(TOPIC), listener)

  while (true) {


    consumer.seekToBeginning(consumer.assignment())
    val records = consumer.poll(20000)
    // for (record <- records.asScala) {
    //   println(record)
    // }
    println("records size "+records.count())
    records.asScala.foreach(rec => println("offset "+rec.offset()))  
  }

}

尝试一下，让我知道。如果你有任何问题。

KafkaConsumer不读取偏移0

1 个答案: