Question

我的Kafka Streams应用程序正在使用以下键值布局的kafka主题使用： String.class -> HistoryEvent.class

在打印我当前的主题时，可以确认：

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic flow-event-stream-file-service-test-instance --property print.key=true --property key.separator=" -- " --from-beginning
flow1 --  SUCCESS     #C:\Daten\file-service\in\crypto.p12

“ flow1”是String键，--之后的部分是序列化值。

我的流程设置如下：

    KStream<String, HistoryEvent> eventStream = builder.stream(applicationTopicName, Consumed.with(Serdes.String(),
            historyEventSerde));


    eventStream.selectKey((key, value) -> new HistoryEventKey(key, value.getIdentifier()))
            .groupByKey()
            .reduce((e1, e2) -> e2,
                    Materialized.<HistoryEventKey, HistoryEvent, KeyValueStore<Bytes, byte[]>>as(streamByKeyStoreName)
                            .withKeySerde(new HistoryEventKeySerde()));

据我所知，我要告诉它使用String和HistoryEvent serde消费该主题，因为这就是该主题中的内容。然后，我将其“重新密钥化”以使用组合密钥，该组合密钥应使用为HistoryEventKey.class提供的Serde本地存储。据我了解，这将导致使用新密钥创建另一个主题（可以在kafka容器中的主题列表中看到）。很好。

现在的问题是，即使在主题只有一个文档的干净环境中，应用程序也无法启动：

org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=0_0, processor=KSTREAM-SOURCE-0000000000, topic=flow-event-stream-file-service-test-instance, partition=0, offset=0
Caused by: org.apache.kafka.streams.errors.StreamsException: A serializer (key: org.apache.kafka.common.serialization.StringSerializer / value: HistoryEventSerializer) is not compatible to the actual key or value type (key type: HistoryEventKey / value type: HistoryEvent). Change the default Serdes in StreamConfig or provide correct Serdes via method parameters.

从消息中很难分辨出问题的确切位置。它在我的基本主题中说，但这是不可能的，因为密钥没有类型HistoryEventKey。由于我在HistoryEventKey中为reduce提供了Serser，所以它也不能在本地商店使用。

对我而言唯一有意义的是，它与selectKey操作有关，该操作会导致重新排列和出现新主题。但是，我无法弄清楚如何为该操作提供Serde。我不想将其设置为默认值，因为它不是默认的密钥Serde。

Answer 1

在对执行进行了更多调试之后，我能够确定在groupByKey步骤中创建了新主题。您可以提供一个Grouped实例，该实例可以指定用于键和值的Serde：

    eventStream.selectKey((key, value) -> new HistoryEventKey(key, value.getIdentifier()))
            .groupByKey(Grouped.<HistoryEventKey, HistoryEvent>as(null)
                    .withKeySerde(new HistoryEventKeySerde())
                    .withValueSerde(new HistoryEventSerde())
            )
            .reduce((e1, e2) -> e2,
                    Materialized.<HistoryEventKey, HistoryEvent, KeyValueStore<Bytes, byte[]>>as(streamByKeyStoreName)
                            .withKeySerde(new HistoryEventKeySerde()));

Answer 2

final StreamsBuilder builder = new StreamsBuilder();
    KStream<String, HistoryEvent> source = builder.stream("test-topic", Consumed.with(Serdes.String(), new HistoryEventSerDes()));
    source.groupBy((key, value) -> HistoryKey.builder()
    .id(value.getKey())
    .build(),
    Grouped.<HistoryKey, HistoryEvent>as("repartition")
    .withKeySerde(new HistoryKeySerDes())
    .withValueSerde(new HistoryEventSerDes()))
    .windowedBy(TimeWindows.of(Duration.ofSeconds(5)))
    .count(Materialized.<HistoryKey, Long, WindowStore<Bytes, byte[]>>as("test-topic-store")
    .withKeySerde(new HistoryKeySerDes())
    .withValueSerde(Serdes.Long()))
    .toStream()
    .print(Printed.toSysOut());

例外

 org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=1_0, processor=KSTREAM-SOURCE-0000000005, topic=test-topic-service-key-repartition-repartition, partition=0, offset=0, stacktrace=org.apache.kafka.streams.errors.StreamsException: A serializer (com.example.org.HistoryKeySerializer) is not compatible to the actual key type (key type: com.example.org.HistoryEvent). Change the default Serdes in StreamConfig or provide correct Serdes via method parameters.
    at org.apache.kafka.streams.state.StateSerdes.rawKey(StateSerdes.java:175)
    at org.apache.kafka.streams.state.internals.MeteredWindowStore.keyBytes(MeteredWindowStore.java:222)
    at org.apache.kafka.streams.state.internals.MeteredWindowStore.fetch(MeteredWindowStore.java:152)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl$WindowStoreReadWriteDecorator.fetch(ProcessorContextImpl.java:539)
    at org.apache.kafka.streams.kstream.internals.KStreamWindowAggregate$KStreamWindowAggregateProcessor.process(KStreamWindowAggregate.java:122)
    at org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:117)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:201)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:180)
    at org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:133)
    at org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:87)
    at org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:363)
    at org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:199)
    at org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:425)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:912)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:819)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:788)
    Caused by: java.lang.ClassCastException: class com.example.org.HistoryEvent cannot be cast to class com.example.org.HistoryKey (com.example.org.HistoryEvent and com.example.org.HistoryKey are in unnamed module of loader org.springframework.boot.loader.LaunchedURLClassLoader @d2cc05a)
    at com.example.org.HistoryKeySerializer.serialize(HistoryKeySerializer.java:13)
    at org.apache.kafka.streams.state.StateSerdes.rawKey(StateSerdes.java:171)
    ... 15 more

重新分区后，Kafka流未使用Serde

2 个答案: