Kafka一遍又一遍地重播消息 - Heartbeat会话过期 - 标记协调员死机

时间:2017-09-29 21:49:54

标签: python apache-kafka kafka-consumer-api

使用python kafka api从主题中读取消息,其中只包含少量消息。 Kafka一遍又一遍地重播队列中的消息。

它从我的主题收到一条消息(随每条消息内容返回),然后抛出kafka.coordinator - ERROR - Heartbeat session expired - marking coordinator dead kafka.coordinator - WARNING - Marking the coordinator dead (node 1) for group GROUPID1: Heartbeat session expired. kafka.coordinator.consumer - WARNING - Auto offset commit failed for group GROUPID1: CommitFailedError: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. kafka.cluster - INFO - Group coordinator for GROUPID1 is BrokerMetadata(nodeId=1, host='HOST', port=PORT, rack=None) kafka.coordinator - INFO - Discovered coordinator 1 for group GROUPID1 kafka.coordinator - INFO - Skipping heartbeat: no auto-assignment or waiting on rebalance kafka.coordinator.consumer - ERROR - Offset commit failed: This is likely to cause duplicate message delivery Traceback (most recent call last): File "/path/python3.5/site-packages/kafka/coordinator/consumer.py", line 407, in _maybe_auto_commit_offsets_sync self.commit_offsets_sync(self._subscription.all_consumed_offsets()) File "/path/python3.5/site-packages/kafka/coordinator/consumer.py", line 398, in commit_offsets_sync raise future.exception # pylint: disable-msg=raising-bad-type kafka.errors.CommitFailedError: CommitFailedError: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. kafka.coordinator.consumer - INFO - Revoking previously assigned partitions {TopicPartition(topic='TOPIC1', partition=0)} for group GROUPID1 并继续循环其余的消息并继续重播它们。更多日志:

filterByUserType

1 个答案:

答案 0 :(得分:2)

好像你需要调整你的消费者配置,很可能看起来似乎消费者的心跳会话即将到期,并且由于会话过期而无法提交最后一次轮询的记录并且它正在触发重新平衡,以便它将从最后一次未提交的记录再次轮询

配置检查 -

  • heartbeat.interval.ms
  • session.timeout.ms
  • max.poll.interval.ms