我是Kafka的新手,正在直播。我正在创建一个本地存储,以保存来自特定主题components
的所有更新。我不是我在做什么错。还有其他从Stream创建商店的方法吗?
我需要在Kafka中创建主题comp-store
吗?
public class MyStream {
final static CountDownLatch latch = new CountDownLatch(1);
private static final String APP_ID = "MyTestApp";
public static void main(String[] args) throws InterruptedException {
final Properties streamsConfiguration = getStreamsConfiguration();
final StreamsBuilder builder = new StreamsBuilder();
//
final KStream<String, Component> componentStream = builder.stream("components");
final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration);
KeyValueMapper<String, Component, Iterable<KeyValue<String, Component>>> mapper = new KeyValueMapper<String, Component, Iterable<KeyValue<String,Component>>>() {
@Override
public Iterable<KeyValue<String, Component>> apply(String list, Component comp) {
ArrayList<KeyValue<String, Component>> result = new ArrayList<>();
result.add(KeyValue.pair(comp.getCompId()+":"+comp.getListId(), comp));
return result;
}
};
KStream<String,Component> componentsStram = componentStream.flatMap(mapper);
KGroupedStream<String,Component> componentsGroupedStream = componentsStram.groupByKey();
componentsGroupedStream.reduce(new Reducer<Component>() {
public Component apply(Component oldVal, Component newVal) {
return newVal;
}
}, Materialized.<String, Component, KeyValueStore<Bytes, byte[]>>as("comp-store"));
streams.start();
new Thread(new Runnable() {
@Override
public void run() {
while(true){
if(streams.state().isRunning()){
latch.countDown();
}
}
}
}).start();
latch.await();
Thread.sleep(5000);
ReadOnlyKeyValueStore<String,Component> localStore = waitUntilStoreIsQueryable("comp-store", QueryableStoreTypes.<String, Component> keyValueStore(), streams);
System.out.println(localStore.approximateNumEntries());
Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
}
private static Properties getStreamsConfiguration() {
Properties settings = new Properties();
settings.put(StreamsConfig.APPLICATION_ID_CONFIG, APP_ID);
settings.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
settings.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass().getName());
settings.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, ProtoSerde.class);
settings.put(StreamsConfig.STATE_DIR_CONFIG, "C:\\temp");
settings.put("auto.offset.reset","earliest");
settings.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
return settings;
}
public static <T> T waitUntilStoreIsQueryable(final String storeName, final QueryableStoreType<T> queryableStoreType, final KafkaStreams streams) throws InterruptedException {
while (true) {
try {
return streams.store(storeName, queryableStoreType);
} catch (InvalidStateStoreException ignored) {
Thread.sleep(100);
}
}
}
}
例外
Exception in thread "main" org.apache.kafka.streams.errors.InvalidStateStoreException: The state store, comp-store, may have migrated to another instance.
at org.apache.kafka.streams.state.internals.QueryableStoreProvider.getStore(QueryableStoreProvider.java:60)
at org.apache.kafka.streams.KafkaStreams.store(KafkaStreams.java:1038)
at com.mr.streams.MyStream.main(MyStream.java:110)
更新 waitUntilStoreIsQueryable
之后,我的异常已解决,但仍然无法查询状态存储。看来它正在无限循环中。但是componentsStram
中有数据。我在这里做错什么了吗?
答案 0 :(得分:2)
由于Kafka Streams实例尚未准备就绪,因此引发了异常。
根据文档:https://docs.confluent.io/current/streams/faq.html#interactive-queries,可能有上述异常的两个原因:
本地KafkaStreams实例尚未准备好,因此无法查询其本地状态存储。
本地KafkaStreams实例已准备就绪,但特定的状态存储区刚刚迁移到幕后的另一个实例。
最简单的处理方法是等待状态存储可查询:
public static <T> T waitUntilStoreIsQueryable(final String storeName,
final QueryableStoreType<T> queryableStoreType,
final KafkaStreams streams) throws InterruptedException {
while (true) {
try {
return streams.store(storeName, queryableStoreType);
} catch (InvalidStateStoreException ignored) {
// store not yet ready for querying
Thread.sleep(100);
}
}
}
更新:
您必须在定义整个拓扑之后移动KafkaStream的创建:
行:final KafkaStreams streams = new KafkaStreams(builder.build(), streamsConfiguration);
应该跟随:componentsGroupedStream.reduce(...)