Question

我正在使用storm和kafka来分析实时数据。

我在spout

中收到以下错误消息

错误

kafka.common.OffsetOutOfRangeException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at java.lang.Class.newInstance(Class.java:374)
at kafka.common.ErrorMapping$.maybeThrowException(ErrorMapping.scala:53)
at kafka.message.ByteBufferMessageSet.kafka$message$ByteBufferMessageSet$$internalIterator(ByteBufferMessageSet.scala:99)
at kafka.message.ByteBufferMessageSet.iterator(ByteBufferMessageSet.scala:82)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:73)
at kafka.message.MessageSet.foreach(MessageSet.scala:87)
at scala.collection.TraversableOnce$class.size(TraversableOnce.scala:104)
at kafka.message.MessageSet.size(MessageSet.scala:87)
at storm.kafka.PartitionManager.fill(PartitionManager.java:113)
at storm.kafka.PartitionManager.next(PartitionManager.java:83)
at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:106)
at backtype.storm.daemon.executor$fn__3430$fn__3445$fn__3474.invoke(executor.clj:547)
at backtype.storm.util$async_loop$fn__444.invoke(util.clj:403)
at clojure.lang.AFn.run(AFn.java:24)
at java.lang.Thread.run(Thread.java:745)

注意：

spout的startOffsetTime为-1。
风暴版 - 0.9.0
kafka verison - 0.7.2

如何解决此问题？

任何建议都将不胜感激。

Answer 1

<强> kafka.common.OffsetOutOfRangeException

通常表示客户已请求服务器上不再提供的范围

这可能发生，因为根据Kafka配置中的保留策略，主题日志中具有请求的偏移量的消息不再存在。

以下是配置示例:(查看并按照最佳设置进行设置）

############################# Log Retention Policy #############################    
# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.    
# The minimum age of a log file to be eligible for deletion
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log as long as the remaining
# segments don't drop below log.retention.bytes.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=536870912

# The interval at which log segments are checked to see if they can be deleted according 
# to the retention policies
log.cleanup.interval.mins=1

注意： Kafka将根据您的配置自动删除文件中的消息，消费者在zookeeper中保留分区偏移量。（现在考虑抵消是3000）。当Kafka完成清理时，将重置此分区的偏移量，以便最大偏移量必须小于存储在zookeeper （3000）中的一个使用者。当消费者从zookeeper获得当前偏移量（即再次为3000）并使用此偏移量来读取不存在的Kafka消息时，这可能是一个问题。因此解决方案是将自动删除间隔处理为最佳。

查看以下链接以获取更多信息。

如何处理OffsetOutOfRangeException错误？

错误

注意：

1 个答案: