经纪人地址更改后,Kafka Producer无法重新连接并刷新Metada

时间:2019-12-05 03:05:03

标签: apache-kafka

我用k8s运行kafka,使用k8s NodePort,在为Kafka客户端获取主题元数据的pod重建之后,它不会更改。外部可以访问Pod Ip和Node Ip。

Kafka版本:2.11-0.11.0.2

测试脚本:

bin/kafka-producer-perf-test.sh \
   --print-metrics \
   --topic $TOPIC \
   --num-records 10000000 \
   --record-size 500 \
   --throughput 10 \
   --producer-props acks=1 metadata.max.age.ms=10000 \
   bootstrap.servers=$KAFKA_0 \
   buffer.memory=67108864 \
   batch.size=8196

输出:

52 records sent, 10.3 records/sec (0.00 MB/sec), 14.6 ms avg latency, 325.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 1.5 ms avg latency, 2.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 1.4 ms avg latency, 2.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 1.3 ms avg latency, 2.0 ms max latency.
51 records sent, 10.2 records/sec (0.00 MB/sec), 1.2 ms avg latency, 2.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 1.1 ms avg latency, 2.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 5.3 ms avg latency, 113.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 1.1 ms avg latency, 2.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 0.9 ms avg latency, 2.0 ms max latency.
[2019-12-05 10:24:41,072] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,128] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,175] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,225] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,403] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,410] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,778] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,807] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:42,567] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:42,680] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:43,417] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:43,709] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

26 records sent, 0.2 records/sec (0.00 MB/sec), 4616.4 ms avg latency, 120000.0 ms max latency.
org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for reconnect-test-2:120000 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for reconnect-test-2:120000 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for reconnect-test-2:120000 ms has passed since batch creation

问题:

After rebuilding the Kafka broker pod, pod IP changed, NodePort not change, the Kafka producer can not reconnect, and print the warning log forever. But there is no problem after restart the test script.

0 个答案:

没有答案