Question

我正在尝试简单的Spring XD应用程序来加载HDFS中的日志事件。我已使用spring-ampq/rabbit log4j appender（org.springframework.amqp.rabbit.log4j.AmqpAppender类）配置目标应用程序，以将日志消息泵送到预配置的交换机。我设置了以下流来从中提取这些消息并将它们推送到HDFS，其中soruce和sink模块都是现成的XD模块，

流定义，

xd:>stream create --name demoQ1 --definition "rabbit | hdfs --rollover=15 --directory=/user/root" --deploy

创建并部署了新的流'demoQ1'

xd:>stream list
  Stream Name  Stream Definition                                   Status
  -----------  --------------------------------------------------  --------
  demoQ1       rabbit | hdfs --rollover=15 --directory=/user/root  deployed

AMQP Appender正在发布消息以交换并将其路由到demoQ1队列，其中兔子来源拾取第一条消息然后卡住，因为它不会确认消息。可能是什么原因？

Answer 1

在容器日志中，您是否看到：“无法将消息有效负载写入HDFS”？

如果是这样，那么您需要使用模块之间的类型转换。从Rabbit源到hdfs接收器，消息将只是字节数组。

您的流定义可以是，

stream create --name demoQ1 --definition“rabbit --outputType = text / plain | hdfs --rollover = 15 --directory = / user / root”--deploy

，或者

stream create --name demoQ1 --definition“rabbit | hdfs --inputType = text / plain --rollover = 15 --directory = / user / root”--deploy

分别注意source / sink中的outputType或inputType选项。在这种情况下，hdfs接收器的HdfsStoreMessageHandler期望有效负载的类型为String。

有关类型转换的详细信息，请查看以下内容： https://github.com/spring-projects/spring-xd/wiki/Type-Conversion

Answer 2

在运行Rabbit模块的spring XD容器上启用调试日志，显示第一条消息反复发生以下异常，并且消息被重新排队，因此消息保持未确认状态，并且兔子源无法处理更多消息..

要解决此问题，请从log4j Appender属性中删除此属性log4j.appender.amqp.contentEncoding=null。此属性显式指定编码器的名称为“null”，这似乎是一个错误。我期待null意味着没有指定编码器：）

日志中的异常，不断重复，因为邮件被拒绝并重新排队等等。

19:29:17,713 DEBUG SimpleAsyncTaskExecutor-1 listener.BlockingQueueConsumer:268 - Received message: (Body:'Hello'MessageProperties [headers={categoryName=org.apache.hadoop.yarn.server.nodemanager.NodeManager, level=INFO}, timestamp=Sat Apr 19 19:21:52 PDT 2014, messageId=null, userId=null, appId=NodeManager, clusterId=null, type=null, correlationId=null, replyTo=null, contentType=text/plain, contentEncoding=null, contentLength=0, deliveryMode=PERSISTENT, expiration=null, priority=0, redelivered=true, receivedExchange=test-exch, receivedRoutingKey=rk1, deliveryTag=184015, messageCount=0]) 19:29:17,715 WARN SimpleAsyncTaskExecutor-1 listener.SimpleMessageListenerContainer:530 - Execution of Rabbit message listener failed, and no ErrorHandler has been set. org.springframework.amqp.rabbit.listener.ListenerExecutionFailedException: Listener threw exception at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.wrapToListenerExecutionFailedExceptionIfNeeded(AbstractMessageListenerContainer.java:751) at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:690) at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:583) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$001(SimpleMessageListenerContainer.java:75) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$1.invokeListener(SimpleMessageListenerContainer.java:154) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.invokeListener(SimpleMessageListenerContainer.java:1111) at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:556) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:904) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:888) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$500(SimpleMessageListenerContainer.java:75) at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:989) at java.lang.Thread.run(Thread.java:722) Caused by: org.springframework.amqp.support.converter.MessageConversionException: failed to convert text-based Message content at org.springframework.amqp.support.converter.SimpleMessageConverter.fromMessage(SimpleMessageConverter.java:100) at org.springframework.integration.amqp.inbound.AmqpInboundChannelAdapter$1.onMessage(AmqpInboundChannelAdapter.java:73) at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:688) ... 10 more Caused by: java.io.UnsupportedEncodingException: null at java.lang.StringCoding.decode(StringCoding.java:190) at java.lang.String.(String.java:416) at java.lang.String.(String.java:481) at org.springframework.amqp.support.converter.SimpleMessageConverter.fromMessage(SimpleMessageConverter.java:97) ... 12 more 19:29:17,715 DEBUG SimpleAsyncTaskExecutor-1 listener.BlockingQueueConsumer:657 - Rejecting messages (requeue=true)

spring XD Rabbit源模块无法处理消息，第一条消息保持未确认状态

2 个答案: