使用Kafka消息总线时，来自AbstractAggregatingMessageGroupProcessor的NullPointerException

时间：2015-12-11 01:11:15

标签： spring apache-kafka spring-xd

非常确定这是Spring XD中的一个错误。

在SingleNode模式下运行Spring XD 1.3.0.RELEASE。除了使用Kafka而不是本地传输之外，所有配置都是默认配置。相关的XD配置：

spring:
  profiles: singlenode
xd:
  transport: kafka
  messagebus:
    kafka:
      brokers:                                 localhost:9092
      zkAddress:                               localhost:2181
      mode:                                    embeddedHeaders
      offsetManagement:                        kafkaTopic
      socketBufferSize:                        2097152
      offsetStoreTopic:                        SpringXdOffsets
      offsetStoreSegmentSize:                  25000000
      offsetStoreRetentionTime:                60000
      offsetStoreRequiredAcks:                 1
      offsetStoreMaxFetchSize:                 1048576
      offsetStoreBatchBytes:                   16384
      offsetStoreBatchTime:                    1000
      offsetUpdateTimeWindow:                  10000
      offsetUpdateCount:                       0
      offsetUpdateShutdownTimeout:             2000
      default:
        batchSize:                 16384
        batchTimeout:              0
        replicationFactor:         1
        concurrency:               1
        requiredAcks:              1
        compressionCodec:          none
        queueSize:                 8192 # must be a power of 2
        maxWait:                   100
        fetchSize:                 1048576
        minPartitionCount:         1
        durableSubscription:       false

使用聚合器创建流（这一直接来自参考文档）：

stream create --name aggregates --definition "http | aggregator --count=3 --aggregation=T(org.springframework.util.StringUtils).collectionToDelimitedString(#this.![payload],' ') | log" --deploy

然后发送3个POST：

xd:> http post --data Hello
xd:> http post --data World
xd:> http post --data !

结果就是这个堆栈跟踪：

2015-12-10T17:07:11-0800 1.3.0.RELEASE ERROR pool-13-thread-1 listener.LoggingErrorHandler - Error while processing: KafkaMessage [Message(magic = 0, attributes = 0, crc = 344940496, key = null, payload = java.nio.HeapByteBuffer[pos=0 lim=81 cap=81]), KafkaMessageMetadata [offset=2, nextOffset=3, Partition[topic='aggregates.0', id=0]]
org.springframework.messaging.MessageHandlingException: error occurred in message handler [org.springframework.integration.config.AggregatorFactoryBean#0]; nested exception is java.lang.NullPointerException
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:139) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.dispatcher.AbstractDispatcher.tryOptimizedDispatch(AbstractDispatcher.java:116) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.dispatcher.UnicastingDispatcher.doDispatch(UnicastingDispatcher.java:147) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.dispatcher.UnicastingDispatcher.dispatch(UnicastingDispatcher.java:120) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.channel.AbstractSubscribableChannel.doSend(AbstractSubscribableChannel.java:77) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:442) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.channel.AbstractMessageChannel.send(AbstractMessageChannel.java:392) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
    at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutput(AbstractMessageProducingHandler.java:231) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.handler.AbstractMessageProducingHandler.produceOutput(AbstractMessageProducingHandler.java:154) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.handler.AbstractMessageProducingHandler.sendOutputs(AbstractMessageProducingHandler.java:102) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:105) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.channel.FixedSubscriberChannel.send(FixedSubscriberChannel.java:69) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.channel.FixedSubscriberChannel.send(FixedSubscriberChannel.java:63) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:115) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
    at org.springframework.messaging.core.GenericMessagingTemplate.doSend(GenericMessagingTemplate.java:45) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
    at org.springframework.messaging.core.AbstractMessageSendingTemplate.send(AbstractMessageSendingTemplate.java:105) ~[spring-messaging-4.2.2.RELEASE.jar:4.2.2.RELEASE]
    at org.springframework.integration.endpoint.MessageProducerSupport.sendMessage(MessageProducerSupport.java:105) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter.access$300(KafkaMessageDrivenChannelAdapter.java:43) ~[spring-integration-kafka-1.3.0.RELEASE.jar:na]
    at org.springframework.integration.kafka.inbound.KafkaMessageDrivenChannelAdapter$AutoAcknowledgingChannelForwardingMessageListener.doOnMessage(KafkaMessageDrivenChannelAdapter.java:171) ~[spring-integration-kafka-1.3.0.RELEASE.jar:na]
    at org.springframework.integration.kafka.listener.AbstractDecodingMessageListener.onMessage(AbstractDecodingMessageListener.java:50) ~[spring-integration-kafka-1.3.0.RELEASE.jar:na]
    at org.springframework.integration.kafka.listener.QueueingMessageListenerInvoker$KafkaMessageDispatchingSubscriber.onNext(QueueingMessageListenerInvoker.java:221) [spring-integration-kafka-1.3.0.RELEASE.jar:na]
    at org.springframework.integration.kafka.listener.QueueingMessageListenerInvoker$KafkaMessageDispatchingSubscriber.onNext(QueueingMessageListenerInvoker.java:209) [spring-integration-kafka-1.3.0.RELEASE.jar:na]
    at reactor.core.processor.util.RingBufferSubscriberUtils.route(RingBufferSubscriberUtils.java:67) [reactor-core-2.0.5.RELEASE.jar:na]
    at reactor.core.processor.RingBufferProcessor$BatchSignalProcessor.run(RingBufferProcessor.java:789) [reactor-core-2.0.5.RELEASE.jar:na]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_66]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_66]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
Caused by: java.lang.NullPointerException: null
    at org.springframework.integration.aggregator.AbstractAggregatingMessageGroupProcessor.aggregateHeaders(AbstractAggregatingMessageGroupProcessor.java:115) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.aggregator.AbstractAggregatingMessageGroupProcessor.processMessageGroup(AbstractAggregatingMessageGroupProcessor.java:79) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.completeGroup(AbstractCorrelatingMessageHandler.java:648) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.aggregator.AbstractCorrelatingMessageHandler.handleMessageInternal(AbstractCorrelatingMessageHandler.java:405) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:127) ~[spring-integration-core-4.2.2.RELEASE.jar:na]
    ... 32 common frames omitted

这是一个Spring XD错误吗？有解决方法吗？

更多细节。

导致NPE的标头是kafka_messageKey。此标头的值为null，因此AbstractAggregatingMessageGroupProcessor:115会抛出NPE。

进一步展望KafkaMessageBus，似乎有意将Kafka messageKey设置为从制作人处为空。

2 个答案:

答案 0 :(得分：2)

一种可能的解决方法是通过在$XD_HOME/modules/processor/aggregator/config/aggregator.xml的开头插入一个标头过滤器，将标头过滤器添加到聚合器模块，方法是更改​​其开头，如下所示：

<channel id="input" />

<channel id="aggregatorInput"/>

<int:header-filter input-channel="inputChannel"
    output-channel="aggregatorInput" header-names="kafka_messageKey"/>

<aggregator input-channel="aggregatorInput" output-channel="output"
            correlation-strategy-expression="${correlation}"
            release-strategy-expression="${release}" expression="${aggregation}"
            send-partial-result-on-expiry="true" expire-groups-upon-completion="true"
            message-store="messageStore">
    </aggregator>

... rest of the module definition remains unchanged ...

至于如何在日志运行中修复此问题，我将评论JIRA问题。

干杯， 的Marius

答案 1 :(得分：1)

这是Spring XD中的一个错误。有关详细信息，请参阅INT-3908。

在Marius的suggestion之后，以下是一个合适的解决方法：

修改$XD_HOME/modules/processor/aggregator/config/aggregator.xml以包含：

<channel id="aggregatorInput"/>

<header-enricher input-channel="input" output-channel="aggregatorInput" default-overwrite="true">
    <header name="kafka_messageKey" value="."/>
</header-enricher>

<aggregator input-channel="aggregatorInput" output-channel="output"
    correlation-strategy-expression="${correlation}"
    release-strategy-expression="${release}" expression="${aggregation}"
    send-partial-result-on-expiry="true" expire-groups-upon-completion="true"
    message-store="messageStore">
</aggregator>

注意：使用header-filter将无效，因为SI用于removing a header的机制仅为remove the header if it is not already null。

或者，如果您不想直接编辑模块XML，可以使用Module Composition在标准聚合器模块之前包含Header Enricher。

module compose --name kafa-aggregator --definition "header-enricher --headers={\"kafka_messageKey\":\"'.'\"} --overwrite=true | aggregator --count=3 --aggregation=T(org.springframework.util.StringUtils).collectionToDelimitedString(#this.![payload],' ')"

stream create --name aggregates --definition "http | kafa-aggregator | log" --deploy
相关问题
最新问题