我用k8s运行kafka,使用k8s NodePort,在为Kafka客户端获取主题元数据的pod重建之后,它不会更改。外部可以访问Pod Ip和Node Ip。
Kafka版本:2.11-0.11.0.2
测试脚本:
bin/kafka-producer-perf-test.sh \
--print-metrics \
--topic $TOPIC \
--num-records 10000000 \
--record-size 500 \
--throughput 10 \
--producer-props acks=1 metadata.max.age.ms=10000 \
bootstrap.servers=$KAFKA_0 \
buffer.memory=67108864 \
batch.size=8196
输出:
52 records sent, 10.3 records/sec (0.00 MB/sec), 14.6 ms avg latency, 325.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 1.5 ms avg latency, 2.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 1.4 ms avg latency, 2.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 1.3 ms avg latency, 2.0 ms max latency.
51 records sent, 10.2 records/sec (0.00 MB/sec), 1.2 ms avg latency, 2.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 1.1 ms avg latency, 2.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 5.3 ms avg latency, 113.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 1.1 ms avg latency, 2.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 0.9 ms avg latency, 2.0 ms max latency.
[2019-12-05 10:24:41,072] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,128] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,175] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,225] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,403] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,410] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,778] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:41,807] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:42,567] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:42,680] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:43,417] WARN [Producer clientId=producer-1] Connection to node 14 (/10.197.129.81:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-12-05 10:24:43,709] WARN [Producer clientId=producer-1] Connection to node 12 (/10.197.128.77:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
26 records sent, 0.2 records/sec (0.00 MB/sec), 4616.4 ms avg latency, 120000.0 ms max latency.
org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for reconnect-test-2:120000 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for reconnect-test-2:120000 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for reconnect-test-2:120000 ms has passed since batch creation
问题:
After rebuilding the Kafka broker pod, pod IP changed, NodePort not change, the Kafka producer can not reconnect, and print the warning log forever. But there is no problem after restart the test script.