Kinesis生产者放入一个分片,而消费者收到其他分片的重复

时间:2019-02-13 07:46:20

标签: java amazon-kinesis amazon-kcl amazon-kinesis-kpl

我的带有随机分区键的Kinesis Producer将记录放置在shard0中。我希望从我的使用者的分片0中读取它。

我正在观察的是,对于某些记录,情况并非如此:

理想情况:放入000000000002并从000000000002中读取

2019-02-12 11:18:52.278  INFO 10152 --- [TaskExecutor-60] c.i.i.e.kinesis.KinesisProducer          : Put Result={ShardId: shardId-000000000002,SequenceNumber: 49592292474998443170974666693116643662109230963259080738,}
Shard: shardId-000000000002, PartitionKey: partitionKey-MTALK__22a5e920-014f-45b0-8d27-aa3f511d3738, SequenceNumber: 49592292474998443170974666693116643662109230963259080738

观测值:放入000000000000并从000000000002中读取

Shard: shardId-000000000002, PartitionKey: partitionKey-MTALK__09241131-5f81-44fc-852f-7f6483b5e18b, SequenceNumber: 49592292458897305137635556784925437217617884560152854530

此外,尽管检查点和Trim Horizo​​n正确,但我看到消费者重新启动时再次使用了相同的消息-来自不同堆的相同分区密钥和相同序列号。

Put Result={ShardId: shardId-000000000003,SequenceNumber: 49592278123331569911393613339504152562720511055896772658,}

Consumed as:

Shard: shardId-000000000002, PartitionKey: partitionKey-MTALK__3f1caefc-e072-4854-8cb8-f42af2cb67e2, SequenceNumber: 49592278123331569911393613339504152562720511055896772658
Shard: shardId-000000000003, PartitionKey: partitionKey-MTALK__3f1caefc-e072-4854-8cb8-f42af2cb67e2, SequenceNumber: 49592278123331569911393613339504152562720511055896772658
Shard: shardId-000000000003, PartitionKey: partitionKey-MTALK__3f1caefc-e072-4854-8cb8-f42af2cb67e2, SequenceNumber: 49592278123331569911393613339504152562720511055896772658
  1. 如何切换分片?
  2. 如果相同的分区键是MD5且是模数,如何更改分片
  3. 为什么尽管有检查点,还是出现重复消息

0 个答案:

没有答案