KafkaSender的sendTransactionally在上一个事务进行期间开始新事务

时间:2019-09-20 09:40:49

标签: java apache-kafka

我使用Kafka 2.3.0。 我有一个反应式控制器,应该将记录发送到一个主题。我尝试完全设置一次投放并将这些值添加到我的属性中:

  processing.guarantee: exactly_once
  isolation.level: read_committed
  enable.auto.commit: false
  linger.ms: 100
  producer.linger.ms: 100
  fetch.max.wait.ms: 100
  partition-count: 16
  replication.factor: 1
  max.in.flight.requests.per.connection: 1

我在创建发件人期间设置了transactionalId。有一种方法可以将请求部分发送到主题:

protected Mono<SenderResult<String>> sendToTopic(@NonNull final String correlationId,
        @NonNull final AbstractCommandRequest request) {
    return Flux
        .just(SenderRecord.create(
                new ProducerRecord<>(internalCoreTopic,
                        new ProjectChannelKey(request.getProjectId(), request.getChannelId()), request),
                correlationId))
        .subscriberContext(ctx -> addMDC(request, ctx))
        .flatMap(records -> sender.sendTransactionally(Flux.just(Flux.just(records))))
        .concatMap(r -> r)
        .reduce((a, b) -> {
            if (b.exception() != null) {
                return b;
            }
            return a;
        });
} 

它可以工作,但是发送者不会等到之前的事务被提交。而且我有时会收到这种例外情况:

 12:01:43.482 [1ingress-forwarders-tridproject-internal-core-2] ERROR c.l.k.m.c.i.rest.IngressController corr.id=009d0d8a-7e58-4f04-a93d-5540047db732/1568970103473:4 - TransactionalId 1ingress-forwarders-tridproject-internal-core: Invalid transition attempted from state IN_TRANSACTION to state IN_TRANSACTION
    org.apache.kafka.common.KafkaException: TransactionalId 1ingress-forwarders-tridproject-internal-core: Invalid transition attempted from state IN_TRANSACTION to state IN_TRANSACTION
        at org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:759)
        at org.apache.kafka.clients.producer.internals.TransactionManager.transitionTo(TransactionManager.java:752)
        at org.apache.kafka.clients.producer.internals.TransactionManager.beginTransaction(TransactionManager.java:217)
        at org.apache.kafka.clients.producer.KafkaProducer.beginTransaction(KafkaProducer.java:643)
        at reactor.kafka.sender.internals.DefaultKafkaSender$DefaultTransactionManager.lambda$null$0(DefaultKafkaSender.java:485)
        at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:53)
        at reactor.core.publisher.MonoLift.subscribe(MonoLift.java:45)
        at reactor.core.publisher.MonoOnAssembly.subscribe(MonoOnAssembly.java:61)
        at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:150)
        at reactor.core.publisher.FluxOnAssembly$OnAssemblySubscriber.onNext(FluxOnAssembly.java:353)
        at com.project.common.config.mdc.MdcContextLifter.onNext(MdcContextLifter.java:33)
        at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:178)
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:50)
        at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:27)
        at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
        at java.util.concurrent.FutureTask.run(FutureTask.java)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
    Assembly trace from producer [reactor.core.publisher.MonoLift] :
        reactor.core.publisher.Mono.create(Mono.java:183)
        reactor.kafka.sender.internals.DefaultKafkaSender$DefaultTransactionManager.lambda$begin$1(DefaultKafkaSender.java:484)
    Error has been observed by the following operator(s):
        |_  Mono.create ⇢ reactor.kafka.sender.internals.DefaultKafkaSender$DefaultTransactionManager.lambda$begin$1(DefaultKafkaSender.java:484)
        |_  Mono.flatMap ⇢ reactor.kafka.sender.internals.DefaultKafkaSender$DefaultTransactionManager.begin(DefaultKafkaSender.java:484)
        |_  Mono.then ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.transaction(TopicDescriptor.java:211)
        |_  Mono.concatWith ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.transaction(TopicDescriptor.java:212)
        |_  Flux.thenMany ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.transaction(TopicDescriptor.java:213)
        |_  Flux.thenMany ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.transaction(TopicDescriptor.java:213)
        |_  Flux.concatWith ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.transaction(TopicDescriptor.java:214)
        |_  Flux.concatWith ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.transaction(TopicDescriptor.java:216)
        |_  Flux.concatWith ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.transaction(TopicDescriptor.java:217)
        |_  Flux.concatWith ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.transaction(TopicDescriptor.java:218)
        |_  Mono.error ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.lambda$transaction$13(TopicDescriptor.java:232)
        |_  Flux.onErrorResume ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.transaction(TopicDescriptor.java:226)
        |_  Flux.publishOn ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.transaction(TopicDescriptor.java:239)
        |_  Flux.concatMapDelayError ⇢ com.project.kafka.queues.TopicDescriptor$KafkaSenderWrapper.sendTransactionally(TopicDescriptor.java:154)
        |_  Flux.concatMap ⇢ com.project.kafka.queues.rest.AbstractKafkaRestController.sendToTopic(AbstractKafkaRestController.java:73)
        |_  Flux.reduce ⇢ com.project.kafka.queues.rest.AbstractKafkaRestController.sendToTopic(AbstractKafkaRestController.java:74)
        |_  Mono.map ⇢ com.project.kafka.ms.client.ingress.rest.IngressController.sendCommandRequestToInternalCore(IngressController.java:263)
        |_  Mono.then ⇢ com.project.kafka.ms.client.ingress.rest.IngressController.lambda$execCommand$7(IngressController.java:130)
        |_  Mono.flatMap ⇢ com.project.kafka.ms.client.ingress.rest.IngressController.execCommand(IngressController.java:103)
        |_  Mono.cast ⇢ com.project.kafka.ms.client.ingress.rest.IngressController.execCommand(IngressController.java:133)

如何强制事务等待上一次事务提交或中止?

0 个答案:

没有答案