建议使用Avro消息使用Kafka实现发件箱模式

时间:2019-06-14 06:52:25

标签: apache-kafka spring-cloud spring-cloud-stream outbox-pattern

发件箱模式要求发件箱表存储实体的有效负载,以便消息中继可以访问它并通过Kafka发送它。我发现的几乎每个示例都以JSON格式存储有效负载:

enter image description here

enter image description here

重点是,我更喜欢使用Avro格式而不是JSON来利用诸如架构演变之类的功能。

下面您将看到一个Avro模式及其生成的Java类:

{
  "namespace" : "com.codependent.outboxpattern.account",
  "type" : "record",
  "name" : "TransferEmmitted",
  "fields" : [
    {"name":"transferId","type":"string"},
    {"name":"sourceAccountId","type":"long"},
    {"name":"destinationAccountId","type":"long"},
    {"name":"ammount","type":"float"}
  ]
}
---
public class TransferEmmitted extends org.apache.avro.specific.SpecificRecordBase implements org.apache.avro.specific.SpecificRecord {
...

1)我应该如何在发件箱表的有效负载中存储TransferEmmited实体?

2)关于消息中继,它应如何从数据库中存储的有效负载中生成一条消息,以在Spring Cloud Stream Kafka生产者中进行传输?

我已经实现了first approach,但似乎有点尴尬,我不确定这是最好的方法。

帐户服务:使用AvroSchemaRegistryClientMessageConverter生成一条Message并提取其字节数组有效载荷,将TransferEmmitted Avro实体保存在发件箱表中:

@Transactional
@Service
class OutboxServiceImpl(private val outboxRepository: OutboxRepository,
                        private val avroMessageConverter: AvroSchemaRegistryClientMessageConverter) : OutboxService {

    override fun save(messageId: String, topic: String, entity: SpecificRecordBase) {
        val message = avroMessageConverter.toMessage(entity, MessageHeaders(mutableMapOf(MessageHeaders.CONTENT_TYPE to "application/*+avro" as Any))) as Message<*>
        outboxRepository.save(Outbox(0, messageId, topic, entity.schema.name, State.PENDING, message.payload as ByteArray))
    }

}

消息中继服务:从发件箱表中读取并发送消息(如果未决):

@Component
class MessageRelay(private val outboxService: OutboxService,
                   private val source: Source) {

    @Scheduled(fixedDelay = 10000)
    fun checkOutbox() {
        val pending = outboxService.getPending()
        pending.forEach {
            val message = MessageBuilder.withPayload(it.payload)
                    .setHeader(KafkaHeaders.MESSAGE_KEY, it.messageKey)
                    .build()
            source.output().send(message)
            outboxService.markAsProcessed(it.id)
        }
    }

}

此实现确实有效,但是有些方面使我对其正确性表示怀疑:

1)帐户服务使用AvroSchemaRegistryClientMessageConverter存储实体的有效负载。也许有更好的方法...

2)从消息中继服务发送到Kafka的消息的内容类型为application/*+avro,而不是application/vnd.transferemitted.v1+avro之类的内容类型。

0 个答案:

没有答案