如何设置信息键?

时间:2019-06-20 15:24:35

标签: spring-cloud spring-cloud-stream

我正在将Spring Cloud与Apache Kafka一起使用。发送到kafka主题的消息通常还包含一个密钥。如何设置该键?

比方说,我产生这样的消息

@StreamListener(CashflowSink.T1_CASHFLOW_IN)
@SendTo(CashflowSink.T2_CASHFLOW_OUT)
public synchronized Cashflow receive1(String message) {
    System.out.println("******************");
    System.out.println("At Sink1");
    System.out.println("******************");
    System.out.println("Received message " + message);

    String arr[] = message.split(";");
    if (arr[0].equalsIgnoreCase("Cashflow")) {
        Cashflow cf = new Cashflow();
        cf.setContractId(Integer.parseInt(arr[1]));
        cf.setDate(arr[2]);
        cf.setAmount(Float.parseFloat(arr[3]));
        return cf;
    }

return null;

}

稍后,我想将现金流量与其他主题结合在一起。对于联接,我需要一个钥匙。假设关键是现金流量的contractId。如何在消息中设置该键?

编辑1

试图合并Gary的代码,我想到了:

    //@Bean //--> leads to: Parameter 0 of method runner in tki.bigdata.steams.CashflowService required a single bean, but 10 were found:
    @Qualifier("t2_cashflow_in")  // is this a correct way to get a handle to the channel?
    public ApplicationRunner runner(MessageChannel output) {
        ((AbstractMessageChannel) output).addInterceptor(0, new ChannelInterceptor() {

            @Override
            public Message<?> preSend(Message<?> message, MessageChannel channel) {
                return MessageBuilder.fromMessage(message)
                        .setHeader(KafkaHeaders.MESSAGE_KEY, ((Cashflow) message.getPayload()).getContractId())
                        .build();
            }

        });
        return args -> {            
        };
    }

    @StreamListener(CashflowSink.T2_CASHFLOW_IN)
    public synchronized void receive2(Cashflow cashflow, @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) byte[] key) {
        System.out.println("******************");
        System.out.println("At Sink2");
        System.out.println("******************");
        System.out.println("Received cashflow " + cashflow);
        System.out.println(cashflow + ", key:" + new String(key));
    }

@Bean注释给我一个错误,可能是因为有多个“通道”。我遵循了错误消息中的提示,并使用了@Channel批注。不确定,这是否满足我的要求。

执行该代码,出现类似以下的错误。所以我认为拦截没有用?

2019-06-20 20:56:02,557 ERROR [Ljava.lang.String;@4195105b.container-0-C-1 o.s.k.l.LoggingErrorHandler:37 - Error while processing: ConsumerRecord(topic = tier2.cashflow, partition = 0, offset = 11, CreateTime = 1561056959426, serialized key size = -1, serialized value size = 63, headers = RecordHeaders(headers = [RecordHeader(key = deliveryAttempt, value = [49]), RecordHeader(key = contentType, value = [34, 97, 112, 112, 108, 105, 99, 97, 116, 105, 111, 110, 47, 106, 115, 111, 110, 34]), RecordHeader(key = spring_json_header_types, value = [123, 34, 100, 101, 108, 105, 118, 101, 114, 121, 65, 116, 116, 101, 109, 112, 116, 34, 58, 34, 106, 97, 118, 97, 46, 117, 116, 105, 108, 46, 99, 111, 110, 99, 117, 114, 114, 101, 110, 116, 46, 97, 116, 111, 109, 105, 99, 46, 65, 116, 111, 109, 105, 99, 73, 110, 116, 101, 103, 101, 114, 34, 44, 34, 99, 111, 110, 116, 101, 110, 116, 84, 121, 112, 101, 34, 58, 34, 106, 97, 118, 97, 46, 108, 97, 110, 103, 46, 83, 116, 114, 105, 110, 103, 34, 125])], isReadOnly = false), key = null, value = [B@5884f4e2)
org.springframework.messaging.MessageHandlingException: Missing header 'kafka_receivedMessageKey' for method parameter type [class [B]
    at org.springframework.messaging.handler.annotation.support.HeaderMethodArgumentResolver.handleMissingValue(HeaderMethodArgumentResolver.java:105)
    at org.springframework.messaging.handler.annotation.support.AbstractNamedValueMethodArgumentResolver.resolveArgument(AbstractNamedValueMethodArgumentResolver.java:106)
    at org.springframework.messaging.handler.invocation.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:117)
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:148)
    at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:116)
    at org.springframework.cloud.stream.binding.StreamListenerMessageHandler.handleRequestMessage(StreamListenerMessageHandler.java:55)
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:123)
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:176)
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:115)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:132)
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:105)
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:73)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:453)
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:401)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:187)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:166)
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:47)
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:109)
    at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:205)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.sendMessageIfAny(KafkaMessageDrivenChannelAdapter.java:369)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$400(KafkaMessageDrivenChannelAdapter.java:74)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:431)
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$IntegrationRecordMessageListener.onMessage(KafkaMessageDrivenChannelAdapter.java:402)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.lambda$onMessage$0(RetryingMessageListenerAdapter.java:120)
    at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287)
    at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:211)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:114)
    at org.springframework.kafka.listener.adapter.RetryingMessageListenerAdapter.onMessage(RetryingMessageListenerAdapter.java:40)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeOnMessage(KafkaMessageListenerContainer.java:1263)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeOnMessage(KafkaMessageListenerContainer.java:1256)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeRecordListener(KafkaMessageListenerContainer.java:1217)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.doInvokeWithRecords(KafkaMessageListenerContainer.java:1198)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeRecordListener(KafkaMessageListenerContainer.java:1118)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.invokeListener(KafkaMessageListenerContainer.java:933)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.pollAndInvoke(KafkaMessageListenerContainer.java:749)
    at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:698)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:745)

如果有更多有用的信息,我会发布整个课程

@EnableBinding(CashflowService.CashflowSink.class)
public class CashflowService {

    @StreamListener(CashflowSink.T1_CASHFLOW_IN)
    @SendTo(CashflowSink.T2_CASHFLOW_OUT)
    public synchronized Cashflow receive1(String message) {
        System.out.println("******************");
        System.out.println("At Sink1");
        System.out.println("******************");
        System.out.println("Received message " + message);

        String arr[] = message.split(";");
        if (arr[0].equalsIgnoreCase("Cashflow")) {
            Cashflow cf = new Cashflow();
            cf.setContractId(Integer.parseInt(arr[1]));
            cf.setDate(arr[2]);
            cf.setAmount(Float.parseFloat(arr[3]));

            return cf;
        }

        return null;

    }

    // @Bean //--> leads to: Parameter 0 of method runner in
    // tki.bigdata.steams.CashflowService required a single bean, but 10 were
    // found:
    @Qualifier("t2_cashflow_in") // is this a correct way to get a handle to the
                                    // channel?
    public ApplicationRunner runner(MessageChannel output) {
        ((AbstractMessageChannel) output).addInterceptor(0, new ChannelInterceptor() {

            @Override
            public Message<?> preSend(Message<?> message, MessageChannel channel) {
                return MessageBuilder.fromMessage(message)
                        .setHeader(KafkaHeaders.MESSAGE_KEY, ((Cashflow) message.getPayload()).getContractId()).build();
            }

        });
        return args -> {
        };
    }

    @StreamListener(CashflowSink.T2_CASHFLOW_IN)
    public synchronized void receive2(Cashflow cashflow, @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) byte[] key) {
        System.out.println("******************");
        System.out.println("At Sink2");
        System.out.println("******************");
        System.out.println("Received cashflow " + cashflow);
        System.out.println(cashflow + ", key:" + new String(key));
    }

    public interface CashflowSink {
        String T1_CASHFLOW_IN = "t1_cashflow_in";
        String T2_CASHFLOW_IN = "t2_cashflow_in";
        String T1_CASHFLOW_OUT = "t1_cashflow_out";
        String T2_CASHFLOW_OUT = "t2_cashflow_out";

        @Input(T1_CASHFLOW_IN)
        SubscribableChannel t1_cashflow_in();

        @Input(T2_CASHFLOW_IN)
        SubscribableChannel t2_cashflow_in();

        @Output(T1_CASHFLOW_OUT)
        SubscribableChannel t1_cashflow_out();

        @Output(T2_CASHFLOW_OUT)
        SubscribableChannel t2_cashflow_out();
    }
}

1 个答案:

答案 0 :(得分:0)

有一个messageKeyExpression kafka扩展生产者属性;问题是,在转换了有效负载之后会对表达式求值(除非您使用nativeEncoding,在这种情况下,您可以将其与payload.contractId一起使用)。

如果您未使用本机编码,则可以添加一个拦截器(在转换拦截器之前),并将密钥提升到KafkaHeaders.MESSAGE_KEY头。

这里是一个例子:

@SpringBootApplication
@EnableBinding(Processor.class)
public class So56689257Application {

    public static void main(String[] args) {
        SpringApplication.run(So56689257Application.class, args);
    }

    @Bean
    public ApplicationRunner runner(MessageChannel output) {
        ((AbstractMessageChannel) output).addInterceptor(0, new ChannelInterceptor() {

            @Override
            public Message<?> preSend(Message<?> message, MessageChannel channel) {
                return MessageBuilder.fromMessage(message)
                        .setHeader(KafkaHeaders.MESSAGE_KEY, ((Foo) message.getPayload()).getContractId().getBytes())
                        .build();
            }

        });
        return args -> {
            output.send(MessageBuilder.withPayload(new Foo("someContractId")).build());
        };
    }

    @StreamListener(Processor.INPUT)
    public void listen(Foo foo, @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY) byte[] key) {
        System.out.println(foo + ", key:" + new String(key));
    }

    public static class Foo {

        private String contractId;

        public Foo() {
            super();
        }

        public Foo(String contractId) {
            this.contractId = contractId;
        }

        public String getContractId() {
            return this.contractId;
        }

        public void setContractId(String contractId) {
            this.contractId = contractId;
        }

        @Override
        public String toString() {
            return "Foo [contractId=" + this.contractId + "]";
        }

    }

}

Foo [contractId=someContractId], key:someContractId

RabbitMQ绑定器已经具有内置的RabbitExpressionEvaluatingInterceptor(用于在有效负载转换之前评估表达式),但是Kafka绑定器目前没有等效项,因此您必须添加自己的拦截器。

或者,如果您使用的是json,则可以在表达式中使用#jsonpath来提取密钥。