我们正在使用kafka（0.9.0.0）来编排不同微服务之间的命令消息。我们发现了一个间歇性的问题，即重复的消息被传递到特定主题。发生此问题时出现的日志如下所示。有人可以帮助理解这个问题
Wed, 21-Sep-2016 09:19:07 - WARNING Coordinator unknown during heartbeat -- will retry
Wed, 21-Sep-2016 09:19:07 - WARNING Heartbeat failed; retrying
Wed, 21-Sep-2016 09:19:07 - WARNING <BrokerConnection host=AZSG-D-BOT-DEV4 port=9092> timed out after 40000 ms. Closing connection.
Wed, 21-Sep-2016 09:19:07 - ERROR Fetch to node 1 failed: RequestTimedOutError - 7 - This error is thrown if the request exceeds the user-specified time limit in the request.
Wed, 21-Sep-2016 09:19:07 - INFO Marking the coordinator dead (node 1): None.
Wed, 21-Sep-2016 09:19:07 - INFO Group coordinator for kafka-python-default-group is BrokerMetadata(nodeId=1, host=u'AZSG-D-BOT-DEV4', port=9092)
Wed, 21-Sep-2016 09:19:07 - ERROR OffsetCommit failed for group kafka-python-default-group due to group error (UnknownMemberIdError - 25 - Returned from group requests (offset commits/fetches, heartbeats, etc) when the memberId is not in the current generation.), will rejoin
Wed, 21-Sep-2016 09:19:07 - WARNING Offset commit failed: group membership out of date This is likely to cause duplicate message delivery.
Wed, 21-Sep-2016 09:19:07 - ERROR LeaveGroup request failed: UnknownMemberIdError - 25 - Returned from group requests (offset commits/fetches, heartbeats, etc) when the memberId is not in the current generation.
Wed, 21-Sep-2016 09:19:07 - INFO Marking the coordinator dead (node 1): None.
Wed, 21-Sep-2016 09:19:07 - INFO Group coordinator for kafka-python-default-group is BrokerMetadata(nodeId=1, host=u'AZSG-D-BOT-DEV4', port=9092)
Wed, 21-Sep-2016 09:19:07 - ERROR OffsetCommit failed for group kafka-python-default-group due to group error (UnknownMemberIdError - 25 - Returned from group requests (offset commits/fetches, heartbeats, etc) when the memberId is not in the current generation.), will rejoin
Wed, 21-Sep-2016 09:19:07 - WARNING Offset commit failed: group membership out of date This is likely to cause duplicate message delivery.
Wed, 21-Sep-2016 09:19:10 - INFO Joined group 'kafka-python-default-group' (generation 5) with member_id kafka-python-1.0.2-8585f310-cb4f-493a-a98d-12ec9810419b
Wed, 21-Sep-2016 09:19:10 - INFO Updated partition assignment: [TopicPartition(topic=u'ilinaTestPlatformReq', partition=0)]
答案 0 :(得分：2)
来自Kafka documentation on Consumer config：
session.timeout.ms（默认为30000） - 用于检测的超时 使用Kafka集团管理设施时失败。当一个 在会话超时内没有收到消费者的心跳 经纪人会将消费者标记为失败并重新平衡该组。以来 只有在调用poll()时才会发送心跳，这是一个更高的会话 超时允许更多时间在消费者的民意调查中进行消息处理 循环以更长的时间来检测硬故障。也可以看看max.poll.records用于控制处理时间的另一个选项 民意调查循环。请注意，该值必须在允许的范围内group.min.session.timeout.ms在代理配置中配置 和group.max.session.timeout.ms。
似乎如果消息处理时间大于30000毫秒，则会触发消费者重新平衡，这可能会导致重复的消息传递。
您可以尝试增加session.timeout.ms。
另一种选择是在处理消息之前使用pause()异步处理消息，在处理消息后使用resume()。在这种情况下，即使处理时间超过poll()，消费者也会调用session.timeout.ms（并发送心跳）。因此，经纪人不会将您的消费者标记为失败，并且不会启动重新平衡。