Question

我编写了一个带有spring cloud aws binder的示例应用程序

<svg viewBox="0 0 400 400">
  <defs>
    <radialGradient id="template">
      <stop offset="10%" stop-color="gold"/>
      <stop offset="95%" stop-color="green"/>
    </radialGradient>
    <radialGradient id="gradient1" gradientTransform="translate(0.25, 0.25) scale(0.5)" href="#template"/>
  </defs>

  <circle fill="url(#template)" cx="60" cy="60" r="50"/>
  <circle fill="url(#gradient1)" cx="160" cy="60" r="50"/>
</svg>

代码

  compile('org.springframework.cloud:spring-cloud-starter-stream-kinesis:1.0.0.BUILD-SNAPSHOT')

application.yml

@StreamListener(Processor.INPUT)
public void receive(Message<String> message) {      
    System.out.println("Message recieved: "+message);
    System.out.println("Message Payload: "+message.getPayload());    

}

我已经在多个端口上启动了应用程序

spring:
  cloud:
    stream:
      bindings:
        input: 
          group: group
          destination: stream
          content-type: application/json          
        output:  
          group: group
          destination: stream
          content-type: application/json

当我将消息发布到流时，大多数情况下多个实例正在使用消息。

例如我发送了消息{“22”：“11”}，这已被8083和8084消费

申请时

消息：8084

8081,8082,8083, 8084.

应用程序上的

消息：8083

2018-03-16 12:29:19.715  INFO 10084 --- [           main] a.i.k.KinesisMessageDrivenChannelAdapter : started KinesisMessageDrivenChannelAdapter{shardOffsets=[KinesisShardOffset{iteratorType=TRIM_HORIZON, sequenceNumber='null', timestamp=null, stream='stream', shard='shardId-000000000000', reset=false}], consumerGroup='group'}
2018-03-16 12:29:19.809  INFO 10084 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8084 (http) with context path ''
2018-03-16 12:29:19.809  INFO 10084 --- [           main] com.example.aws.AwsApplication           : Started AwsApplication in 21.034 seconds (JVM running for 22.975)
2018-03-16 12:29:19.840  INFO 10084 --- [esis-consumer-1] a.i.k.KinesisMessageDrivenChannelAdapter : The [ShardConsumer{shardOffset=KinesisShardOffset{iteratorType=TRIM_HORIZON, sequenceNumber='null', timestamp=null, stream='stream', shard='shardId-000000000000', reset=false}, state=NEW}] has been started.
2018-03-16 12:30:23.929  INFO 10084 --- [esis-consumer-1] a.i.k.KinesisMessageDrivenChannelAdapter : The records '[{SequenceNumber: 49582549849562056887358041088912873574803531055853731842,ApproximateArrivalTimestamp: Fri Mar 16 12:30:21 IST 2018,Data: java.nio.HeapByteBuffer[pos=0 lim=47 cap=47],PartitionKey: partitionKey-0,}]' are skipped from processing because their sequence numbers are less than already checkpointed: 49582549849562056887358041088912873574803531055853731842
Message recieved: GenericMessage [payload={"22":"11"}, headers={aws_shard=shardId-000000000000, id=f6cb4b6d-e149-059f-7e4d-aa9dfeeef10e, contentType=application/json, aws_receivedStream=stream, aws_receivedPartitionKey=partitionKey-0, aws_receivedSequenceNumber=49582549849562056887358041088914082500623155992949948418, timestamp=1521183774995}]
Message Payload: {"22":"11"}

理想情况下，组中只有一个消费者应处理消息。我在这里错过了什么。

Answer 1

感谢您验证解决方案！

我想我发现了问题所在。它位于ShardCheckpointer.checkpoint(String sequenceNumber)。

目前的代码是这样的：

if (existingSequence == null ||
        new BigInteger(existingSequence).compareTo(new BigInteger(sequenceNumber)) < 0) {
        this.checkpointStore.put(this.key, sequenceNumber);
        return true;
}

当存在竞争条件时，两个（所有？）节点检查状态并从商店获得较小的值。所以，我们正在通过条件然后我们都去了checkpointStore.put()部分。在这里，所有这些都存储一个新的相同值并返回true以让Channel Adapter处理相同的记录。

我对此的解决方法如下：

       if (existingSequence == null ||
                new BigInteger(existingSequence).compareTo(new BigInteger(sequenceNumber)) < 0) {
            if (existingSequence != null) {
                return this.checkpointStore.replace(this.key, existingSequence, sequenceNumber);
            }
            else {
                this.checkpointStore.put(this.key, sequenceNumber);
                return true;
            }
        }

相同的条件，但我也使用checkpointStore.replace()与此合同：

/**
 * Atomically replace the value for the key in the store if the old
 * value matches the oldValue argument.

现在我尝试提出测试用例进行验证，并在BUILD-SNAPSHOT准备好消费并验证您的时候通知您。

Spring cloud aws stream，消息由消费者组中的多个实例使用

1 个答案: