Kafka:SASL_SSL + ACL可以产生但不能消耗

时间:2019-10-12 18:49:00

标签: apache-kafka kafka-consumer-api

使用omega,我可以使用用户kafka-console-producer将消息发布到主题acl 使用write,我无法以用户kafka-console-consumer的身份从主题acl读取消息

但是,我可以登录,所有ACL都是正确的,当我使用错误的密码时会抱怨,因此SASL_SSL和ACL可以正常工作。在read中,启用kafka-authorizer.log模式后:

DEBUG

[2019-10-12 20:33:08,647] DEBUG operation = Read on resource = Topic:LITERAL:acl from host = XXXXXXXX is Allow based on acl = User:read has Allow permission for operations: All from hosts: * (kafka.authorizer.logger) [2019-10-12 20:33:08,647] DEBUG Principal = User:read is Allowed Operation = Describe from host = XXXXXXXX on resource = Topic:LITERAL:acl (kafka.authorizer.logger) [2019-10-12 20:33:08,652] DEBUG operation = Read on resource = Group:LITERAL:aclRead from host = XXXXXXXX is Allow based on acl = User:read has Allow permission for operations: Read from hosts: * (kafka.authorizer.logger) [2019-10-12 20:33:08,652] DEBUG Principal = User:read is Allowed Operation = Describe from host = XXXXXXXX on resource = Group:LITERAL:aclRead (kafka.authorizer.logger) 内显示:

kafka-request.log

这基本上意味着一切都很好。

我已经打开kafdrop安装并能够连接到kafka集群。我可以看到那里的所有内容,从主题到消息(!)。但是它说没有消费者与此主题相关。

当我关闭消费者时,它说[2019-10-12 20:40:33,587] DEBUG Completed request:RequestHeader(apiKey=API_VERSIONS, apiVersion=2, clientId=read, correlationId=1) -- {},response:{error_code=0,api_versions=[{api_key=0,min_version=0,max_version=7},{api_key=1,min_version=0,max_version=11},{api_key=2,min_version=0,max_version=5},{api_key=3,min_version=0,max_version=8},{api_key=4,min_version=0,max_version=2},{api_key=5,min_version=0,max_version=1},{api_key=6,min_version=0,max_version=5},{api_key=7,min_version=0,max_version=2},{api_key=8,min_version=0,max_version=7},{api_key=9,min_version=0,max_version=5},{api_key=10,min_version=0,max_version=2},{api_key=11,min_version=0,max_version=5},{api_key=12,min_version=0,max_version=3},{api_key=13,min_version=0,max_version=2},{api_key=14,min_version=0,max_version=3},{api_key=15,min_version=0,max_version=3},{api_key=16,min_version=0,max_version=2},{api_key=17,min_version=0,max_version=1},{api_key=18,min_version=0,max_version=2},{api_key=19,min_version=0,max_version=3},{api_key=20,min_version=0,max_version=3},{api_key=21,min_version=0,max_version=1},{api_key=22,min_version=0,max_version=1},{api_key=23,min_version=0,max_version=3},{api_key=24,min_version=0,max_version=1},{api_key=25,min_version=0,max_version=1},{api_key=26,min_version=0,max_version=1},{api_key=27,min_version=0,max_version=0},{api_key=28,min_version=0,max_version=2},{api_key=29,min_version=0,max_version=1},{api_key=30,min_version=0,max_version=1},{api_key=31,min_version=0,max_version=1},{api_key=32,min_version=0,max_version=2},{api_key=33,min_version=0,max_version=1},{api_key=34,min_version=0,max_version=1},{api_key=35,min_version=0,max_version=1},{api_key=36,min_version=0,max_version=1},{api_key=37,min_version=0,max_version=1},{api_key=38,min_version=0,max_version=1},{api_key=39,min_version=0,max_version=1},{api_key=40,min_version=0,max_version=1},{api_key=41,min_version=0,max_version=1},{api_key=42,min_version=0,max_version=1},{api_key=43,min_version=0,max_version=0},{api_key=44,min_version=0,max_version=0}],throttle_time_ms=0} from connection 192.168.1.13:9094-XXXXXXXXXX:45642-4;totalTime:0.733,requestQueueTime:0.055,localTime:0.468,remoteTime:0.0,throttleTime:0.432,responseQueueTime:0.052,sendTime:0.172,securityProtocol:SASL_SSL,principal:User:read,listener:SASL_SSL (kafka.request.logger) [2019-10-12 20:40:33,604] DEBUG Completed request:RequestHeader(apiKey=METADATA, apiVersion=8, clientId=read, correlationId=2) -- {topics=[{name=acl}],allow_auto_topic_creation=true,include_cluster_authorized_operations=false,include_topic_authorized_operations=false},response:{throttle_time_ms=0,brokers=[{node_id=2,host=kafka2.exmaple.com,port=9094,rack=null},{node_id=3,host=kafka3.exmaple.com,port=9094,rack=null},{node_id=1,host=kafka1.exmaple.com,port=9094,rack=null}],cluster_id=TIIhlmDsSv-wfmkf3PQA4w,controller_id=2,topics=[{error_code=0,name=acl,is_internal=false,partitions=[{error_code=0,partition_index=0,leader_id=1,leader_epoch=3,replica_nodes=[1,3],isr_nodes=[3,1],offline_replicas=[]},{error_code=0,partition_index=4,leader_id=2,leader_epoch=1,replica_nodes=[2,3],isr_nodes=[2,3],offline_replicas=[]},{error_code=0,partition_index=1,leader_id=2,leader_epoch=2,replica_nodes=[2,1],isr_nodes=[2,1],offline_replicas=[]},{error_code=0,partition_index=2,leader_id=2,leader_epoch=1,replica_nodes=[3,2],isr_nodes=[2,3],offline_replicas=[]},{error_code=0,partition_index=3,leader_id=1,leader_epoch=2,replica_nodes=[1,2],isr_nodes=[2,1],offline_replicas=[]}],topic_authorized_operations=0}],cluster_authorized_operations=0} from connection 192.168.1.13:9094-XXXXXXXXXXX:45642-4;totalTime:6.546,requestQueueTime:0.085,localTime:1.913,remoteTime:0.0,throttleTime:0.664,responseQueueTime:4.327,sendTime:0.242,securityProtocol:SASL_SSL,principal:User:read,listener:SASL_SSL (kafka.request.logger) [2019-10-12 20:40:33,606] DEBUG Completed request:RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=2, clientId=read, correlationId=0) -- {key=aclRead,key_type=0},response:{throttle_time_ms=0,error_code=0,error_message=NONE,node_id=2,host=kafka2.exmaple.com,port=9094} from connection 192.168.1.13:9094-XXXXXXXXXXXX:45642-4;totalTime:1.463,requestQueueTime:0.047,localTime:1.209,remoteTime:0.0,throttleTime:0.251,responseQueueTime:0.055,sendTime:0.163,securityProtocol:SASL_SSL,principal:User:read,listener:SASL_SSL (kafka.request.logger) 是我使用以下方法启动的:

Processed a total of 0 messages

consumer.properties内容:

bash kafka-console-consumer.sh --bootstrap-server kafka1.example.com:9094 --topic acl --group aclRead --from-beginning --consumer.config=/root/consumer.properties

我的ACL是正确的,否则它拒绝连接:

security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-256
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username='read' password='blablabla';
ssl.truststore.location=/root/kafka.truststore.jks
ssl.truststore.password=blablabla

DEBUG日志文件也证实了这一点,它们似乎都喜欢发生了什么事。

我还可以看到Current ACLs for resource `Group:LITERAL:aclRead`: User:read has Allow permission for operations: All from hosts: * Current ACLs for resource `Topic:LITERAL:acl`: User:read has Allow permission for operations: All from hosts: * 主题中的条目

__consumer_offsets

所以发生了什么事...

但是,是的...。没有消息,帮助!

1 个答案:

答案 0 :(得分:1)

万一有人偶然发现了这个东西:

我在文件/etc/kafka/tools-log4j.properties(CentOS)中启用了调试日志记录

然后,在启动消费者时,它会显示很多信息,包括有关group leader not available的消息。

事实证明,我使用在server.properties文件中提供的错误默认设置启动了我的3经纪人集群。重新安装服务器并进行更改后,它可以正常工作! 请注意,我仍在开发中,以确保一切正常运行,显然,当第一个使用者连接时,将使用此设置。

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3

以上设置在1文件中将server.properties作为其默认值,这在三经纪人设置期间破坏了使用者。