Question

在仅使用一个Kafka经纪人的测试设置中启动我们的Kafka Streams应用程序时，我们看到以下错误，大约15次运行中有1次发生

org.apache.kafka.streams.errors.StreamsException: Existing internal topic alarm-message-streams-by-organization-repartition has invalid partitions: expected: 32; actual: 12. Use 'kafka.tools.StreamsResetter' tool to clean up invalid topics before processing.

当我们看到实际分区数以上的错误有所不同时（预期为32，实际分区大于0且低于32）。

我们正在执行org.apache.kafka.streams.KafkaStreams#cleanUp，然后再调用org.apache.kafka.streams.KafkaStreams#start。每次测试运行时，Kafka代理都会启动时没有数据（使用https://hub.docker.com/r/wurstmeister/kafka/）。

在查看Kafka经纪人的日志时，我们看到以下内容：

2018-10-22 18:41:31,373] INFO Topic creation Map(
    alarm-message-streams-by-organization-repartition-19 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-22 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-0 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-7 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-23 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-1 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-24 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-2 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-30 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-5 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-21 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-8 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-14 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-15 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-6 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-16 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-31 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-25 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-9 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-20 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-29 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-13 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-26 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-17 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-4 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-10 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-3 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-11 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-12 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-28 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-27 -> ArrayBuffer(42),
    alarm-message-streams-by-organization-repartition-18 -> ArrayBuffer(42)
) (kafka.zk.AdminZkClient)

看起来该主题是用预期的分区数（32）创建的。稍后，在同一日志中，看起来好像有再次创建该主题的请求。我们不知道为什么会这样，但是至少请求仍然包含预期的分区数（32）：

[2018-10-22 18:43:29,851] INFO [Admin Manager on Broker 42]: Error processing create topic request for topic alarm-message-streams-by-organization-repartition with arguments (numPartitions=32, replicationFactor=1, replicasAssignments={}, configs={cleanup.policy=delete, segment.bytes=52428800, segment.ms=600000, retention.ms=9223372036854775807, segment.index.bytes=52428800}) (kafka.server.AdminManager)
org.apache.kafka.common.errors.TopicExistsException: Topic 'alarm-message-streams-by-organization-repartition' already exists.

我们从未与6个Kafka经纪人一起在非测试中发生这种情况。但是，我们运行的测试运行数量要比非测试运行的数量大得多。

注意：导致错误的主题并不总是相同的。

该错误在我们的测试设置中造成了断断续续的感觉，因此我们想了解原因并进行处理。有人能提供一些有关Kafka Streams行为的见解吗？

我们正在使用Kafka和Kafka Streams 2.0.0。

Answer 1

似乎从Kafka集群（即您的单个代理）接收到不完整/不正确的元数据。在启动时（或更准确地说，在每次重新平衡时），Kafka Streams会检查内部主题是否存在预期的分区数。如果主题不存在，则会创建该主题（此主题在应用程序的实时运行期间仅应发生一次）。如果存在正确数量的分区，则使用该主题。如果该主题的分区数量不正确，则会引发您报告的异常。

呼叫{file path}.java:307: error: cannot find symbol @Nullable DataBindingComponent component) { ^ symbol: class MyClass location: class MyBinding在这里不会有任何影响。您可以通过KafkaStreams#cleanup()进行呼叫的StreamResetter（与https://kafka.apache.org/20/documentation/streams/developer-guide/app-reset-tool.html）不同

我目前不知道该问题的根本原因是什么，即，为什么Kafka Streams接收到不正确的主题元数据。希望这会有所帮助。

现有内部主题的分区无效

1 个答案: