我设计了一个Spark Streaming应用程序，该应用程序可在120秒的微型批处理上运行，从本质上将JSON转换为Avro文件。
自从我开始使用新的Kafka主题以来，我遇到了一个问题，我每隔几个微批就随机获得OffsetOutOfRangeException。
这些微型批次非常小，大约有20条消息，只需几秒钟即可完成。
在程序结束时，我commitASync并可以使用OffsetCallBack确认这些提交对Kafka成功。
我能够消除此错误的唯一方法是将消息镜像到另一个Kafka主题（具有与第一个相同的属性）。
使用主主题，我尝试过通过增加Spark资源（执行程序内存，驱动程序内存，YARN队列），禁用Kafka使用者缓存，将Spark升级到2.4.0，减少微批处理时间来解决应用程序级别的问题到30s，60s，增加我的容错设置以允许更多错误
b'2019-04-24 12:24:01,218 INFO - Computing topic xx.xx.xx.xx.xx, partition 0 offsets 1190252 -> 1190271\n'
b'2019-04-24 12:26:00,003 INFO - xx.xx.xx.xx.EntryPoint$$anonfun$main$13$$anon$1: Offset onComplete for - xx.xx.xx.xx.xx- 0 - 1190252\n'
b'2019-04-24 12:26:00,003 INFO - xx.xx.xx.xx.EntryPoint$$anonfun$main$13$$anon$1: Offsets successfully committed\n'
b'2019-04-24 12:26:00,092 INFO - Computing topic xx.xx.xx.xx.xx, partition 0 offsets 1190271 -> 1190295\n'
b'2019-04-24 12:26:00,114 WARN - Lost task 0.0 in stage 8.0 (TID 27, xx, executor 1): org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {xx.xx.xx.xx.xx-0=1190272}\n\tat org.apache.kafka.clients.consumer.internals.Fetcher.parseFetchedData(Fetcher.java:589)\n\tat org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:355)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1000)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:938)\n\tat org.apache.spark.streaming.kafka010.CachedKafkaConsumer.poll(CachedKafkaConsumer.scala:136)\n\tat org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:71)\n\tat org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:271)\n\tat org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:2...
b'2019-04-24 12:26:00,114 WARN - Lost task 0.0 in stage 8.0 (TID 27, xx, executor 1): org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {xx.xx.xx.xx.xx-0=1190272}\n\tat org.apache.kafka.clients.consumer.internals.Fetcher.parseFetchedData(Fetcher.java:589)\n\tat org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:355)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1000)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:938)\n\tat org.apache.spark.streaming.kafka010.CachedKafkaConsumer.poll(CachedKafkaConsumer.scala:136)\n\tat org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:71)\n\tat org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:271)\n\tat org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:2...
b'2019-04-24 12:26:00,119 INFO - Computing topic xx.xx.xx.xx.xx, partition 0 offsets 1190271 -> 1190295\n'
b'2019-04-24 12:26:00,247 WARN - Lost task 0.1 in stage 8.0 (TID 28, xx, executor 1): org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {xx.xx.xx.xx.xx-0=1190271}\n\tat org.apache.kafka.clients.consumer.internals.Fetcher.parseFetchedData(Fetcher.java:589)\n\tat org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:355)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1000)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:938)\n\tat org.apache.spark.streaming.kafka010.CachedKafkaConsumer.poll(CachedKafkaConsumer.scala:136)\n\tat org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:68)\n\tat org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:271)\n\tat org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:2...
b'2019-04-24 12:26:00,247 WARN - Lost task 0.1 in stage 8.0 (TID 28, xx, executor 1): org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {xx.xx.xx.xx.xx-0=1190271}\n\tat org.apache.kafka.clients.consumer.internals.Fetcher.parseFetchedData(Fetcher.java:589)\n\tat org.apache.kafka.clients.consumer.internals.Fetcher.fetchedRecords(Fetcher.java:355)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1000)\n\tat org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:938)\n\tat org.apache.spark.streaming.kafka010.CachedKafkaConsumer.poll(CachedKafkaConsumer.scala:136)\n\tat org.apache.spark.streaming.kafka010.CachedKafkaConsumer.get(CachedKafkaConsumer.scala:68)\n\tat org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:271)\n\tat org.apache.spark.streaming.kafka010.KafkaRDDIterator.next(KafkaRDD.scala:2...
b'2019-04-24 12:26:00,252 INFO - Computing topic xx.xx.xx.xx.xx, partition 0 offsets 1190271 -> 1190295\n'
我几乎排除了在应用程序级别上可以做的任何事情，为什么当我在上次运行中明确提交时，为什么不能从Kafka中读取这些偏移量？