我一直在尝试在Kafka集群上运行我的生产者和消费者API,但它无法运行。
1)Zookeeper正在一个GCP VM实例上运行
2)Kafka正在另一个GCP VM实例上运行
遵循的步骤:
步骤1)使用以下命令运行Zookeeper:
bin/zookeeper-server-start.sh config/zookeeper.properties
Zookeeper.properties:
#the directory where the snapshot is stored.
dataDir=/tmp/zookeeper
#the port at which the clients will connect
clientPort=2181
#disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
# In milliseconds
tickTime=2000
# In ticks
initLimit=10
syncLimit=5
maxClientCnxns=30
# All Zookeeper servers need to be aware of other Zookeepers part of the cluster
server.1=zook-1:2888:3888
注意:zook-1是Zookeeper实例的名称
第2步)
在另一个GCP VM实例上运行Kafka:
bin/kafka-console-producer.sh --broker-li de-kafka-1:9092 --topic test
server.properties:
############################# Server Basics ############################# # The id of the broker. This must be set to a unique integer for each broker.
broker.id=0
############################ Zookeeper ############################# #
# root directory for all kafka znodes.
zookeeper.connect=zook-1:2181
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000
host.name=de-kafka-1
第3步)
从步骤2(使用另一个SSH隧道)在Kafka实例上运行producer api,如下所示:
bin/kafka-console-producer.sh --broker-list de-kafka-1:9 092 --topic test
注意:这里de-kafka-1是我尝试过的GCP Kafka VM实例的名称,用de-Kafka-1实例的外部IP替换了de-kafka-1,localhost,zook-1(Zookeeper的名称) VM实例),zook-1 vm实例的外部IP
Error: OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N Sending msg1 sending msg2
Error: error when sending message to topic test single node with key: null, value: 5 bytes with error (org.apache.kafka.clients.producer.internals.errorloggingcallback)org.apache.kafka.common.errors.timeoutexception: failed to update metadata after 60000 ms.
第4步:
在Kafka实例上运行Consumer API(使用第三个SSH隧道):
bin/kafka-console-consumer.sh --bootstrap-server de-kafka-1:2181 --topic test --from-beginning
如果有人可以提供帮助,我们将不胜感激。感谢