无法连接3个经纪人Kafka

时间:2019-10-22 07:23:03

标签: apache-kafka kerberos kafka-consumer-api spring-kafka kafka-producer-api

我已将连接kafka和kafka的kerberized spring boot应用程序不在我的管理之下。我进行了配置以连接kerberized卡夫卡

#spring.kafka.bootstrap-servers=kafka1:9092,kafka2:9092,kafka3:9092
#appconnect.apm.collector.topic=app_metrics
#spring.kafka.properties.security.protocol=SASL_PLAINTEXT

#spring.kafka.properties.sasl.mechanism=GSSAPI

#spring.kafka.properties.sasl.jaas.config=com.sun.security.auth.module.Krb5LoginModule required debug=true renewTicket=true  serviceName="kafka" principal="principalName@KFK.SECURE" useKeyTab=true keyTab="{path}/to/kafka.keytab";

#spring.kafka.properties.sasl.kerberos.service.name=kafka

我的本​​地测试一切正常。成功生成记录时我记录了日志。我已经在我的本地计算机上进行了测试,并且使用kerberized kafka与1个经纪人一起正常工作。我用远程kafka和另一台在Amazon上的机器进行了测试,它也可以正常工作。

但是,当我将我的spring boot应用程序与他们的3个经纪人kafka共享给客户时,发生了错误,但是我不知道它是什么。请给我一个帮助,我正在共享日志。

还有一件事,我做了一个配置,可以在我的应用程序启动时读取属性,如果kerberos配置为空,它将尝试在不进行身份验证的情况下连接普通的kafka。

客户说有或没有kerberos都没有健康的联系。


更新

我认为它使用kerberos票。

Successfully logged in.
2019-10-17 06:47:08.554 DEBUG [bootstrap,,,] 1 --- [main] o.a.k.c.security.kerberos.KerberosLogin  : [Principal=*********]: It is a Kerberos ticket
2019-10-17 06:47:08.558 TRACE [bootstrap,,,] 1 --- [main] o.a.k.c.s.authenticator.LoginManager     : LoginManager(serviceName=kafka, publicCredentials=[], refCount=1) acquired
2019-10-17 06:47:08.558  INFO [bootstrap,,,] 1 --- [kafka-kerberos-refresh-thread-*********] o.a.k.c.security.kerberos.KerberosLogin  : [Principal=*********]: TGT refresh thread started.
2019-10-17 06:47:08.561 DEBUG [bootstrap,,,] 1 --- [kafka-kerberos-refresh-thread-u*********] o.a.k.c.security.kerberos.KerberosLogin  : Found TGT with client principal '*********' and server principal 'krbtgt/*********'.
2019-10-17 06:47:08.562  INFO [bootstrap,,,] 1 --- [kafka-kerberos-refresh-thread-*********] o.a.k.c.security.kerberos.KerberosLogin  : [Principal=*********]: TGT valid starting at: Thu Oct 17 06:47:08 UTC 2019
2019-10-17 06:47:08.562  INFO [bootstrap,,,] 1 --- [kafka-kerberos-refresh-thread-*********] o.a.k.c.security.kerberos.KerberosLogin  : [Principal=*********]: TGT expires: Thu Oct 17 16:47:08 UTC 2019
2019-10-17 06:47:08.562  INFO [bootstrap,,,] 1 --- [kafka-kerberos-refresh-**********] o.a.k.c.security.kerberos.KerberosLogin  : [Principal=**************]: TGT refresh sleeping until: Thu Oct 17 15:13:58 UTC 2019

我的应用程序已启动,但仍然出现以下错误。


Removing nodeFound least loaded node kafka3:9092 (id: 2 rack: null)
Initialize
Initiating connection to node kafka3:9092 (id: 2 rack: null) using address kafka3/()
Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2
Completed connection to node 2. Fetching API versions.
Initiating API versions fetch from node 2.
No version information found when sending API_VERSIONS with correlation id 797 to node 2. Assuming version 2.
Sending API_VERSIONS {} with correlation id 797 to node 2
Removing node kafka3:9092 (id: 2 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 1
Removing node kafka1:9092 (id: 0 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0
Found least loaded node kafka2:9092 (id: 1 rack: null)
Initialize connection to node kafka2:9092 (id: 1 rack: null) for sending metadata request
Initiating connection to node kafka2:9092 (id: 1 rack: null) using address kafka2/(IP here)
Removing node kafka2:9092 (id: 1 rack: null) from least loaded node selection: is-blacked-out: false, in-flight-requests: 0
Found least loaded node kafka:9092 (id: 0 rack: null)
Sending metadata request (type=MetadataRequest, topics=) to node kafka1:9092 (id: 0 rack: null)
Using older server API v5 to send METADATA {topics=[],allow_auto_topic_creation=true} with correlation id 798 to node 0
Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1
Completed connection to node 1. Fetching API versions.
Initiating API versions fetch from node 1.
No version information found when sending API_VERSIONS with correlation id 799 to node 1. Assuming version 2.
Sending API_VERSIONS {} with correlation id 799 to node 1
Completed receive from node 0 for METADATA with correlation id 798, received {throttle_time_ms=0,brokers=[{node_id=2,host=kafka3=9092,rack=null},Updated cluster metadata version 409 to MetadataCache{cluster=Cluster(id = 8Tcl7QjQS16ldE0OXEZwGw, nodes = [kafka1:9092 (id: 0 rack: null), kafka3:9092 (id: 2 rack: null), kafka2:9092 (id: 1 rack: null)], partitions = [], controller = kafka1:9092 (id: 0 rack: null))}
Completed receive from node 2 for API_VERSIONS with correlation id 797, received {error_code=35,api_versions=[]}
Initiating API versions fetch from node 2.
No version information found when sending API_VERSIONS with correlation id 800 to node 2. Assuming version 0.
Using older server API v0 to send API_VERSIONS {} with correlation id 800 to node 2
Completed receive from node 2 for API_VERSIONS with correlation id 800, received {error_code=0,api_versions=[{api_key=0,min_version=0,max_version=5},{api_key=1,min_version=0,max_version=6}
2019-10-18 14:22:06.760 DEBUG [bootstrap,,,] 1 --- [kafka-producer-network-thread | producer-1] org.apache.kafka.clients.NetworkClient   : 
[Producer clientId=producer-1] Recorded API versions for node 1: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0 to 6 [usable: 6], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0 to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], 
OffsetCommit(8): 0 to 3 [usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10): 0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12): 0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14): 0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1], ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2 [usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21): 0 [usable: 0],
 InitProducerId(22): 0 [usable: 0], OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0 [usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0 [usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0 [usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 [usable: 0], AlterConfigs(33): 0 [usable: 0], AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0 [usable: 0],
 SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): UNSUPPORTED, RenewDelegationToken(39): UNSUPPORTED,
 ExpireDelegationToken(40): UNSUPPORTED, DescribeDelegationToken(41): UNSUPPORTED, DeleteGroups(42): UNSUPPORTED, ElectPreferredLeaders(43): UNSUPPORTED)

所有日志都和上面一样连续。.

有什么想法吗?

它与kerberos有关吗?与较旧的Kafka客户端版本有关吗?网络连接是否正常,我不确定,我也不知道

谢谢

0 个答案:

没有答案