我有一个Kafka Streams应用程序,我正在其中加入一个读取以下内容的KStream 从“ topic1”中读取一个从“ topic2”中读取的GlobalKTable,然后 另一个从“ topic3”读取的GlobalKTable。
当我尝试同时将消息推送到所有3个主题时,就会出现以下异常-
org.apache.kafka.streams.errors.InvalidStateStoreException
如果我在这些主题中一一推送消息,即在主题2中,然后在主题3中,然后在主题1中推送消息,那么我不会得到此消息
在启动KafkaStreams之前,我还添加了StateListener
KafkaStreams.StateListener stateListener = new KafkaStreams.StateListener() {
@Override
public void onChange (KafkaStreams.State newState, KafkaStreams.State oldState) {
if(newState == KafkaStreams.State.REBALANCING) {
try {
Thread.sleep(1000);
}
catch (InterruptedException e) {
e.printStackTrace();
}
}
}
};
streams.setStateListener(stateListener);
streams.start();
我还等到流开始后通过调用以下方法来查询商店
public static <T> T waitUntilStoreIsQueryable(final String storeName,
final QueryableStoreType<T> queryableStoreType,
final KafkaStreams streams) throws InterruptedException {
while (true) {
try {
return streams.store(storeName, queryableStoreType);
} catch (final InvalidStateStoreException ignored) {
// store not yet ready for querying
Thread.sleep(100);
}
}
}
以下是Kafka Streams和GlobalKTable连接代码:
KStream<String, GenericRecord> topic1KStream =
builder.stream(
"topic1",
Consumed.with(Serdes.String(), genericRecordSerde)
);
GlobalKTable<String, GenericRecord> topic2KTable =
builder.globalTable(
"topic2",
Consumed.with(Serdes.String(), genericRecordSerde),
Materialized.<String, GenericRecord, KeyValueStore<Bytes, byte[]>>as("topic2-global-store")
.withKeySerde(Serdes.String())
.withValueSerde(genericRecordSerde)
);
GlobalKTable<String, GenericRecord> topic3KTable =
builder.globalTable(
"topic3",
Consumed.with(Serdes.String(), genericRecordSerde),
Materialized.<String, GenericRecord, KeyValueStore<Bytes, byte[]>>as("topic3-global-store")
.withKeySerde(Serdes.String())
.withValueSerde(genericRecordSerde)
);
KStream<String, MergedObj> stream_topic1_topic2 = topic1KStream.join(
topic2KTable,
(topic2Id, topic1Obj) -> topic1.get("id").toString(),
(topic1Obj, topic2Obj) -> new MergedObj(topic1Obj, topic2Obj)
);
final KStream<String, GenericRecord> enrichedStream =
stream_topic1_topic2.join(
topic3KTable,
(topic2Id, mergedObj) -> mergedObj.topic3Id(),
(mergedObj, topic3Obj) -> new Enriched(
mergedObj.topic1Obj,
mergedObj.topic2Obj,
topic3Obj
).enrich()
);
enrichedStream.to("enrichedStreamTopic", Produced.with(Serdes.String(),getGenericRecordSerde()));
上面的代码与this非常相似。
当我尝试将消息同时推送到所有3个主题时,我得到 以下例外:
org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=0_1, processor=KSTREAM-SOURCE-0000000000, topic=topic1,
partition=1, offset=61465,
stacktrace=org.apache.kafka.streams.errors.InvalidStateStoreException:
Store topic2-global-store is currently closed.
at
org.apache.kafka.streams.state.internals.WrappedStateStore.validateStoreOpen(WrappedStateStore.java:66)
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.get(CachingKeyValueStore.java:150)
at
org.apache.kafka.streams.state.internals.CachingKeyValueStore.get(CachingKeyValueStore.java:37)
at
org.apache.kafka.streams.state.internals.MeteredKeyValueStore.get(MeteredKeyValueStore.java:135)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl$KeyValueStoreReadOnlyDecorator.get(ProcessorContextImpl.java:245)
at
org.apache.kafka.streams.kstream.internals.KTableSourceValueGetterSupplier$KTableSourceValueGetter.get(KTableSourceValueGetterSupplier.java:49)
at
org.apache.kafka.streams.kstream.internals.KStreamKTableJoinProcessor.process(KStreamKTableJoinProcessor.java:71)
at
org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:117)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:183)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:162)
at
org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:122)
at
org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:87)
at
org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:364)
at
org.apache.kafka.streams.processor.internals.AssignedStreamsTasks.process(AssignedStreamsTasks.java:199)
at
org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:420)
at
org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:890)
at
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:805)
at
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:774)
答案 0 :(得分:0)
我解决了这个问题
在我的代码中,我有auto.register.schemas=false
,因为我已经为所有主题手动注册了架构。
设置auto.register.schemas=true
并重新运行流应用程序后,它运行良好。我认为它的内部主题需要使用此标志。