Spring Integration KafkaProducerMessageHandler有关元数据发送的TimeoutException

时间:2016-01-26 13:15:38

标签: spring-integration apache-kafka

我正在尝试使用Spring Integration和Docker上的Kafka实现以下场景:

  • 轮询HTTP端点,结果为String
  • 将结果发送给Kafka

轮询工作正常,我的应用程序设置似乎是正确的,因为应用程序启动时没有任何连接异常。

正如您在输出中看到的那样,KafkaProducerMessageHandler接收消息,但挂起“尝试向节点-1发送元数据请求”超过一分钟,然后抛出TimeoutException。

以下是链接所需的输出和代码:

OUTPUT

kafka-broker.log

zookeeper.log

这是我的应用程序设置:

docker-compose.yml

application.yml

KafkaIntegration.java

KafkaConfig基于https://github.com/spring-projects/spring-integration-samples的Kafka示例和https://github.com/wurstmeister/kafka-docker的Kafka Docker容器。

感谢您的任何想法和建议!!!!

2 个答案:

答案 0 :(得分:1)

我们在您的日志中看到类似的内容:

227 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : Sending metadata request ClientRequest(expectResponse=true, payload=null, request=RequestSend(header={api_key=3,api_version=0,correlation_id=23,client_id=producer-1}, body={topics=[test.topic]})) to node -1
2016-01-26 15:22:24.986 DEBUG 4227 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : Trying to send metadata request to node -1

看起来你真的应该查看Broker日志。

首先它可能是:

kafka.common.InvalidTopicException: topic name TOPIC= is illegal, contains a character other than ASCII alphanumerics, '.', '_' and '-'

https://issues.apache.org/jira/browse/KAFKA-1884

因此,主题名称中的.是非法的。您可以将不兼容的Kafka版本与其客户端一起使用。 Spring Integration Kafka 1.3基于Kafka-0.8.2.2。

从另一方面来看,可能存在一些网络问题:Kafka 0.8.2.2 - Unable to publish messages

答案 1 :(得分:1)

我是个傻瓜!

我的码头组成是:

kafka:
    image: wurstmeister/kafka
    ports:
        - "9092:9092"
    links:
        - zookeeper:zk
    environment:
        KAFKA_ADVERTISED_HOST_NAME: 192.168.59.103
    volumes:
        - /var/run/docker.sock:/var/run/docker.sock

其中192.168.59.103是我的IP。我不得不把它改成localhost,现在看起来很好。

17:25:26.797 DEBUG 25436 --- [ask-scheduler-9] o.s.integration.channel.DirectChannel    : preSend on channel 'kafkaChannel', message: GenericMessage [payload={"login":"markuslamm","id":1075906,"avatar_url":"https://avatars.githubusercontent.com/u/1075906?v=3","gravatar_id":"","url":"https://api.github.com/users/markuslamm","html_url":"https://github.com/markuslamm","followers_url":"https://api.github.com/users/markuslamm/followers","following_url":"https://api.github.com/users/markuslamm/following{/other_user}","gists_url":"https://api.github.com/users/markuslamm/gists{/gist_id}","starred_url":"https://api.github.com/users/markuslamm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markuslamm/subscriptions","organizations_url":"https://api.github.com/users/markuslamm/orgs","repos_url":"https://api.github.com/users/markuslamm/repos","events_url":"https://api.github.com/users/markuslamm/events{/privacy}","received_events_url":"https://api.github.com/users/markuslamm/received_events","type":"User","site_admin":false,"name":"Markus Lamm","company":null,"blog":null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":21,"public_gists":0,"followers":0,"following":0,"created_at":"2011-09-24T07:04:34Z","updated_at":"2016-01-26T15:19:00Z"}, headers={Server=GitHub.com, Cache-Control=public, max-age=60, s-maxage=60, ETag="c06cdef5b902bb019cced50412d4b913", Vary=[Accept, Accept-Encoding], Last-Modified=1453821540000, id=6bffbe99-0e6d-55a5-65a1-208c7be7542d, Content-Length=1135, contentType=application/json;charset=utf-8, http_statusCode=200, Date=1453825526000, timestamp=1453825526797}]
2016-01-26 17:25:26.797 DEBUG 25436 --- [ask-scheduler-9] ssor$ReplyProducingMessageHandlerWrapper : kafkaIntegration.kafkaHandler.serviceActivator.handler received message: GenericMessage [payload={"login":"markuslamm","id":1075906,"avatar_url":"https://avatars.githubusercontent.com/u/1075906?v=3","gravatar_id":"","url":"https://api.github.com/users/markuslamm","html_url":"https://github.com/markuslamm","followers_url":"https://api.github.com/users/markuslamm/followers","following_url":"https://api.github.com/users/markuslamm/following{/other_user}","gists_url":"https://api.github.com/users/markuslamm/gists{/gist_id}","starred_url":"https://api.github.com/users/markuslamm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markuslamm/subscriptions","organizations_url":"https://api.github.com/users/markuslamm/orgs","repos_url":"https://api.github.com/users/markuslamm/repos","events_url":"https://api.github.com/users/markuslamm/events{/privacy}","received_events_url":"https://api.github.com/users/markuslamm/received_events","type":"User","site_admin":false,"name":"Markus Lamm","company":null,"blog":null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":21,"public_gists":0,"followers":0,"following":0,"created_at":"2011-09-24T07:04:34Z","updated_at":"2016-01-26T15:19:00Z"}, headers={Server=GitHub.com, Cache-Control=public, max-age=60, s-maxage=60, ETag="c06cdef5b902bb019cced50412d4b913", Vary=[Accept, Accept-Encoding], Last-Modified=1453821540000, id=6bffbe99-0e6d-55a5-65a1-208c7be7542d, Content-Length=1135, contentType=application/json;charset=utf-8, http_statusCode=200, Date=1453825526000, timestamp=1453825526797}]
2016-01-26 17:25:26.797 DEBUG 25436 --- [ask-scheduler-9] o.s.i.k.o.KafkaProducerMessageHandler    : kafkaHandler received message: GenericMessage [payload={"login":"markuslamm","id":1075906,"avatar_url":"https://avatars.githubusercontent.com/u/1075906?v=3","gravatar_id":"","url":"https://api.github.com/users/markuslamm","html_url":"https://github.com/markuslamm","followers_url":"https://api.github.com/users/markuslamm/followers","following_url":"https://api.github.com/users/markuslamm/following{/other_user}","gists_url":"https://api.github.com/users/markuslamm/gists{/gist_id}","starred_url":"https://api.github.com/users/markuslamm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markuslamm/subscriptions","organizations_url":"https://api.github.com/users/markuslamm/orgs","repos_url":"https://api.github.com/users/markuslamm/repos","events_url":"https://api.github.com/users/markuslamm/events{/privacy}","received_events_url":"https://api.github.com/users/markuslamm/received_events","type":"User","site_admin":false,"name":"Markus Lamm","company":null,"blog":null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":21,"public_gists":0,"followers":0,"following":0,"created_at":"2011-09-24T07:04:34Z","updated_at":"2016-01-26T15:19:00Z"}, headers={Server=GitHub.com, Cache-Control=public, max-age=60, s-maxage=60, ETag="c06cdef5b902bb019cced50412d4b913", Vary=[Accept, Accept-Encoding], Last-Modified=1453821540000, id=6bffbe99-0e6d-55a5-65a1-208c7be7542d, Content-Length=1135, contentType=application/json;charset=utf-8, http_statusCode=200, Date=1453825526000, timestamp=1453825526797}]
2016-01-26 17:25:26.798 DEBUG 25436 --- [ask-scheduler-9] ssor$ReplyProducingMessageHandlerWrapper : handler 'kafkaIntegration.kafkaHandler.serviceActivator.handler' produced no reply for request Message: GenericMessage [payload={"login":"markuslamm","id":1075906,"avatar_url":"https://avatars.githubusercontent.com/u/1075906?v=3","gravatar_id":"","url":"https://api.github.com/users/markuslamm","html_url":"https://github.com/markuslamm","followers_url":"https://api.github.com/users/markuslamm/followers","following_url":"https://api.github.com/users/markuslamm/following{/other_user}","gists_url":"https://api.github.com/users/markuslamm/gists{/gist_id}","starred_url":"https://api.github.com/users/markuslamm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markuslamm/subscriptions","organizations_url":"https://api.github.com/users/markuslamm/orgs","repos_url":"https://api.github.com/users/markuslamm/repos","events_url":"https://api.github.com/users/markuslamm/events{/privacy}","received_events_url":"https://api.github.com/users/markuslamm/received_events","type":"User","site_admin":false,"name":"Markus Lamm","company":null,"blog":null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":21,"public_gists":0,"followers":0,"following":0,"created_at":"2011-09-24T07:04:34Z","updated_at":"2016-01-26T15:19:00Z"}, headers={Server=GitHub.com, Cache-Control=public, max-age=60, s-maxage=60, ETag="c06cdef5b902bb019cced50412d4b913", Vary=[Accept, Accept-Encoding], Last-Modified=1453821540000, id=6bffbe99-0e6d-55a5-65a1-208c7be7542d, Content-Length=1135, contentType=application/json;charset=utf-8, http_statusCode=200, Date=1453825526000, timestamp=1453825526797}]
2016-01-26 17:25:26.798 DEBUG 25436 --- [ask-scheduler-9] o.s.integration.channel.DirectChannel    : postSend (sent=true) on channel 'kafkaChannel', message: GenericMessage [payload={"login":"markuslamm","id":1075906,"avatar_url":"https://avatars.githubusercontent.com/u/1075906?v=3","gravatar_id":"","url":"https://api.github.com/users/markuslamm","html_url":"https://github.com/markuslamm","followers_url":"https://api.github.com/users/markuslamm/followers","following_url":"https://api.github.com/users/markuslamm/following{/other_user}","gists_url":"https://api.github.com/users/markuslamm/gists{/gist_id}","starred_url":"https://api.github.com/users/markuslamm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markuslamm/subscriptions","organizations_url":"https://api.github.com/users/markuslamm/orgs","repos_url":"https://api.github.com/users/markuslamm/repos","events_url":"https://api.github.com/users/markuslamm/events{/privacy}","received_events_url":"https://api.github.com/users/markuslamm/received_events","type":"User","site_admin":false,"name":"Markus Lamm","company":null,"blog":null,"location":null,"email":null,"hireable":null,"bio":null,"public_repos":21,"public_gists":0,"followers":0,"following":0,"created_at":"2011-09-24T07:04:34Z","updated_at":"2016-01-26T15:19:00Z"}, headers={Server=GitHub.com, Cache-Control=public, max-age=60, s-maxage=60, ETag="c06cdef5b902bb019cced50412d4b913", Vary=[Accept, Accept-Encoding], Last-Modified=1453821540000, id=6bffbe99-0e6d-55a5-65a1-208c7be7542d, Content-Length=1135, contentType=application/json;charset=utf-8, http_statusCode=200, Date=1453825526000, timestamp=1453825526797}]
2016-01-26 17:25:26.799 DEBUG 25436 --- [ask-scheduler-9] o.s.integration.channel.DirectChannel    : postSend (sent=true) on channel 'httpPollChannel', message: GenericMessage [payload=, headers={id=55aecfc4-ea57-d769-4275-801b39a0416d, timestamp=1453825526420}]