无法使用Spring Boot连接到kafka经纪人

时间:2019-10-24 07:13:29

标签: spring-boot apache-kafka kerberos spring-kafka kafka-producer-api

我有spring boot应用程序,我们有3个经纪人kafka,它不在我的管理之下,无法连接。当我在本地单一经纪人1.0.1版本的kafka上尝试应用程序时,它将连接并且可以生产或消费,但是。

他们的kafka版本是1.0.1。我的客户端版本是2.2.0。我什至不知道下面的日志是什么。

我的应用程序启动时,它使用KafkaAdmin并记录一些内容,尝试获取kafka主题等。

当我调查日志时,看到很多error_code = 35,这意味着不支持API版本。 (https://kafka.apache.org/protocol#protocol_error_codes),但就像我说的那样,我可以连接并产生消息。

我知道1.0.1 kafka版本支持kafka 2.2.0客户端版本。我有1.3.6 RELEASE版本的springframework.kafka。

我不确定这是否是版本问题,他们的计算机是Windows,但是服务器在linux计算机上,并且使用docker swarm在dockerized kafka上运行。有可能无法连接吗?但是以下日志显示已连接但不适当。此处(https://github.com/wurstmeister/kafka-docker/issues/456)。可能是他们的听众配置错误,我不清楚,而且我没有机会学习。

他们是否有可能升级了kafka版本,并且没有进行inter.broker.protocol.version配置,因此其经纪人无法通信?还有一件事,我正在使用通过SASL_PLAINTEXT进行通信的kerberized kafka。但是,客户端无法连接带有或不带有kerberos的kafka,因此我跳过了这一点。在这里(Kafka upgrade : Need to specify inter.broker.protocol.version when upgraded?


2019-10-17 08:20:26.796 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Node 2 disconnected.
2019-10-17 08:20:26.796 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Removing node kafka3:9092 (id: 2 rack: null) from least loaded node selection: is-blacked-out: true, in-flight-requests: 0
2019-10-17 08:20:26.796 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Removing node kafka1:9092 (id: 0 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0
2019-10-17 08:20:26.796 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Found least loaded node kafka2:9092 (id: 1 rack: null)
2019-10-17 08:20:26.797 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Initialize connection to node kafka2:9092 (id: 1 rack: null) for sending metadata request
2019-10-17 08:20:26.798 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Initiating connection to node kafka2:9092 (id: 1 rack: null) using address kafka2/10.194.63.156
2019-10-17 08:20:26.799 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] o.apache.kafka.common.network.Selector   : [Producer clientId=producer-1] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1
2019-10-17 08:20:26.799 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Completed connection to node 1. Fetching API versions.
2019-10-17 08:20:26.800 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Initiating API versions fetch from node 1.
2019-10-17 08:20:26.800 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] No version information found when sending API_VERSIONS with correlation id 51 to node 1. Assuming version 2.
2019-10-17 08:20:26.800 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Sending API_VERSIONS {} with correlation id 51 to node 1
2019-10-17 08:20:26.800 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Removing node kafka3:9092 (id: 2 rack: null) from least loaded node selection: is-blacked-out: true, in-flight-requests: 0
2019-10-17 08:20:26.800 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Removing node kafka2:9092 (id: 1 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 1
2019-10-17 08:20:26.800 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Found least loaded node kafka1:9092 (id: 0 rack: null)
2019-10-17 08:20:26.800 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Sending metadata request (type=MetadataRequest, topics=) to node kafka1:9092 (id: 0 rack: null)
2019-10-17 08:20:26.800 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Using older server API v5 to send METADATA {topics=[],allow_auto_topic_creation=true} with correlation id 52 to node 0
2019-10-17 08:20:26.803 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Completed receive from node 1 for API_VERSIONS with correlation id 51, received {error_code=35,api_versions=[]}
2019-10-17 08:20:26.803 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Initiating API versions fetch from node 1.
2019-10-17 08:20:26.803 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] No version information found when sending API_VERSIONS with correlation id 53 to node 1. Assuming version 0.
2019-10-17 08:20:26.803 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Using older server API v0 to send API_VERSIONS {} with correlation id 53 to node 1
2019-10-17 08:20:26.805 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Completed receive from node 1 for API_VERSIONS with correlation id 53, received {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=5},{api_key=1,min_version=0,max_version=6},{api_key=2,min_version=0,max_version=2},{api_key=3,min_version=0,max_version=5},{api_key=4,min_version=0,max_version=1},{api_key=5,min_version=0,max_version=0},{api_key=6,min_version=0,max_version=4},{api_key=7,min_version=0,max_version=1},{api_key=8,min_version=0,max_version=3},{api_key=9,min_version=0,max_version=3},{api_key=10,min_version=0,max_version=1},{api_key=11,min_version=0,max_version=2},{api_key=12,min_version=0,max_version=1},{api_key=13,min_version=0,max_version=1},{api_key=14,min_version=0,max_version=1},{api_key=15,min_version=0,max_version=1},{api_key=16,min_version=0,max_version=1},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=1},{api_key=19,min_version=0,max_version=2},{api_key=20,min_version=0,max_version=1},{api_key=21,min_version=0,max_version=0},{api_key=22,min_version=0,max_version=0},{api_key=23,min_version=0,max_version=0},{api_key=24,min_version=0,max_version=0},{api_key=25,min_version=0,max_version=0},{api_key=26,min_version=0,max_version=0},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=0},{api_key=29,min_version=0,max_version=0},{api_key=30,min_version=0,max_version=0},{api_key=31,min_version=0,max_version=0},{api_key=32,min_version=0,max_version=0},{api_key=33,min_version=0,max_version=0},{api_key=34,min_version=0,max_version=0},{api_key=35,min_version=0,max_version=0},{api_key=36,min_version=0,max_version=0},{api_key=37,min_version=0,max_version=0}]}
2019-10-17 08:20:26.805 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Recorded API versions for node 1: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0 to 6 [usable: 6], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0 to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3 [usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10): 0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12): 0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14): 0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1], ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2 [usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21): 0 [usable: 0], InitProducerId(22): 0 [usable: 0], OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0 [usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0 [usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0 [usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 [usable: 0], AlterConfigs(33): 0 [usable: 0], AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0 [usable: 0], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): UNSUPPORTED, RenewDelegationToken(39): UNSUPPORTED, ExpireDelegationToken(40): UNSUPPORTED, DescribeDelegationToken(41): UNSUPPORTED, DeleteGroups(42): UNSUPPORTED, ElectPreferredLeaders(43): UNSUPPORTED)
2019-10-17 08:20:26.806 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Completed receive from node 0 for METADATA with correlation id 52, received {throttle_time_ms=0,brokers=[{node_id=2,host=kafka3,port=9092,rack=null},{node_id=1,host=2,port=9092,rack=null},{node_id=0,host=kafka1,port=9092,rack=null}],cluster_id=8Tcl7QjQS16ldE0OXEZwGw,controller_id=0,topic_metadata=[]}
2019-10-17 08:20:26.806 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.Metadata        : Updated cluster metadata version 23 to MetadataCache{cluster=Cluster(id = 8Tcl7QjQS16ldE0OXEZwGw, nodes = kafka1:9092 (id: 0 rack: null), kafka2:9092 (id: 1 rack: null), kafka3:9092 (id: 2 rack: null)], partitions = [], controller = kafka1:9092 (id: 0 rack: null))}
2019-10-17 08:25:26.834 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Removing node kafka2:9092 (id: 1 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0
2019-10-17 08:25:26.834 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Removing node kafka3:9092 (id: 2 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0
2019-10-17 08:25:26.834 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Found least loaded node kafka1:9092 (id: 0 rack: null)
2019-10-17 08:25:26.834 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Sending metadata request (type=MetadataRequest, topics=) to node kafka1:9092 (id: 0 rack: null)
2019-10-17 08:25:26.834 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Using older server API v5 to send METADATA {topics=[],allow_auto_topic_creation=true} with correlation id 54 to node 0
2019-10-17 08:25:26.836 TRACE [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : [Producer clientId=producer-1] Completed receive from node 0 for METADATA with correlation id 54, received {throttle_time_ms=0,brokers=[{node_id=2,host=kafka3,port=9092,rack=null},{node_id=1,host=kafka2,port=9092,rack=null},{node_id=0,host=kafka1,port=9092,rack=null}],cluster_id=8Tcl7QjQS16ldE0OXEZwGw,controller_id=0,topic_metadata=[]}
2019-10-17 08:25:26.836 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.Metadata        : Updated cluster metadata version 24 to MetadataCache{cluster=Cluster(id = 8Tcl7QjQS16ldE0OXEZwGw, nodes = [kafka1:9092 (id: 0 rack: null), kafka2:9092 (id: 1 rack: null), kafka3:9092 (id: 2 rack: null)], partitions = [], controller = kafka1:9092 (id: 0 rack: null))}

日志在1分钟内连续不变。在这一点上,我感到非常困惑。

谢谢!

0 个答案:

没有答案