Zoookeer配置

Question

我正在运行3节点Kafka群集（3-Kafka，3-Zookeeper-全部在同一VLAN的不同节点上）。所有最新的Kafka版本- kafka_2.12-2.1.0 （无docker）。

我希望对所有外部客户端进行身份验证和授权，但在Kafka-Kafka或Kafka-Zookeeper之间不需要身份验证/授权。

因此，我使用以下配置设置了所有zookeper和Kafka。

Zoookeer配置

clientPort=2181
maxClientCnxns=0
tickTime=2000
server.1=10.0.ZK.1:2888:3888
server.2=10.0.ZK.2:2888:3888
server.3=10.0.ZK.3:2888:3888
initLimit=20
syncLimit=10
autopurge.snapRetainCount=3
autopurge.purgeInterval=24

Kafka配置

broker.id=0 #same like 1 and 2 on Kafka#2 and Kafka#3
listeners=PLAINTEXT://10.0.KF.1:9092,SASL_PLAINTEXT://10.0.KF.1:9093
advertised.listeners=PLAINTEXT://10.0.KF.1:9092,SASL_PLAINTEXT://10.0.KF.1:9093
# same with Kafka#2 and Kafka#3 with respective IPs - 10.0.KF.2, 10.0.KF.3 

offsets.topic.replication.factor=2
transaction.state.log.replication.factor=2
transaction.state.log.min.isr=1
zookeeper.connect=10.0.ZK.1:2181,10.0.ZK.2:2181,10.0.ZK.3:2181
zookeeper.connection.timeout.ms=6000
group.initial.rebalance.delay.ms=3
auto.create.topics.enable=false
num.replica.fetchers=4
default.replication.factor=2
min.insync.replicas=1
listener.security.protocol.map= PLAINTEXT:PLAINTEXT, SASL_PLAINTEXT:SASL_PLAINTEXT
sasl.enabled.mechanisms=SCRAM-SHA-256
security.inter.broker.protocol=PLAINTEXT
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin;User:ANONYMOUS

JAAS配置-在单独的文件中。

KafkaServer {
    org.apache.kafka.common.security.scram.ScramLoginModule required;
};

创建一个主题（ test.topic ），其复制因子= 2，分区= 1000

在这里，我能够产生/使用消息而没有任何问题。但是在Kafka日志中间歇性地（特别是在清晨）跟踪ERROR / Warnings。

WARN Client session timed out, have not heard from server in 4002ms for sessionid 0x2019d038949000a (org.apache.zookeeper.ClientCnxn)
INFO Client session timed out, have not heard from server in 4002ms for sessionid 0x2019d038949000a, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
WARN SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/opt/kafka/config/kafka_jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
INFO Opening socket connection to server **MASKED**/10.0.ZK.3:2181 (org.apache.zookeeper.ClientCnxn)
ERROR [ZooKeeperClient] Auth failed. (kafka.zookeeper.ZooKeeperClient)

ERROR [ReplicaFetcher replicaId=2, leaderId=1, fetcherId=3] Error for partition test.topic-258 at offset 0 (kafka.server.ReplicaFetcherThread)
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.

WARN [LeaderEpochCache test.topic-281] New epoch entry EpochEntry(epoch=2, startOffset=0) caused truncation of conflicting entries ListBuffer(EpochEntry(epoch=0, startOffset=0)). Cache now contains 1 entries. (kafka.server.epoch.LeaderEpochFileCache)

WARN Client session timed out, have not heard from server in 4002ms for sessionid 0x2019d0389490008 (org.apache.zookeeper.ClientCnxn)

我怀疑上面是由于为内部访问（Kafka-Kafka，Kafka-Zookeeper）和外部访问配置了两种不同的协议。有什么方法可以配置Kafka来避免这些情况？非常感谢您提供有关上述问题的其他帮助。

编辑：我添加了“会话超时”日志，还验证了GC时间-最大值为80ms，因此上述情况并非由于GC +所有服务器位于同一VLAN上，因此也排除了它们之间的主要网络数据包丢失。

Kafka内部和外部配置的不同身份验证/授权配置的连接问题

Zoookeer配置

Kafka配置

JAAS配置-在单独的文件中。

0 个答案: