Consumer.committed(topicAndPartition)变为“空”

时间:2019-05-27 03:14:23

标签: scala apache-spark apache-kafka spark-streaming spark-streaming-kafka

每当spark Streaming应用尝试将偏移量提交给Kafka时,它将变为空。

   val consumer = new KafkaConsumer[String,String](kafka_props)    //create a 2nd consumer to fetch last offset
   import java.util
   consumer.subscribe(util.Arrays.asList("topic_1"))   //Subscribe to the 2nd consumer. Without this step, the offsetAndMetadata can't be fetched.
   val offsetAndMetadata = consumer.committed(topicAndPartition)    //Find last committed offset for the given topicAndPartition
   val endOffset = offsetAndMetadata.offset().toLong   //fetch the last committed offset from offsetAndMetadata and cast it to Long data type.
   val fetch_from_offset = Map(new org.apache.kafka.common.TopicPartition("topic_1", 0) -> endOffset) // create a Map with data type (TopicPartition, Long)

只要到了这一点,它就会变为空

val offsetAndMetadata = Consumer.committed(topicAndPartition)

0 个答案:

没有答案