Question

From a stream (k,v), I want to calculate a stream (k, (v,f)) where f is the frequency of the occurrences of a given key in the last n seconds. Give a topic (t1) if I use a windowed table to calculate the frequency:

KTable<Windowed<Integer>,Long> t1_velocity_table = t1_stream.groupByKey().windowedBy(TimeWindows.of(n*1000)).count();

This will give a windowed table with the frequency of each key.

Assuming I won’t be able to join with a Windowed key, instead of the table above I am mapping the stream to a table with simple key:

t1_Stream.groupByKey()
                .windowedBy(TimeWindows.of( n*1000)).count()
                .toStream().map((k,v)->new KeyValue<>(k.key(), Math.toIntExact(v))).to(frequency_topic);
KTable<Integer,Integer> t1_frequency_table = builder.table(frequency_topic);

If I now lookup in this table when a new key arrives in my stream, how do I know if this lookup table will be updated first or the join will occur first (which will cause the stale frequency to be added in the record rather that the current updated one). Will it be better to create a stream instead of table and then do a windowed join ? I want to lookup the table with something like this:

KStream<Integer,Tuple<Integer,Integer>> t1_enriched = t1_Stream.join(t1_frequency_table, (l,r) -> new Tuple<>(l, r));

So instead of having just a stream of (k,v) I have a stream of (k,(v,f)) where f is the frequency of key k in the last n seconds.

Any thoughts on what would be the right way to achieve this ? Thanks.

Answer 1

对于您共享的特定程序，将首先处理流端记录。原因是，您通过主题管道数据......

处理记录时，它将更新聚合结果，该结果将发出写入直通主题的更新记录。之后，记录将由连接运算符处理。之后，新的poll()调用最终将从通过主题读取聚合结果并更新连接的表格侧。

使用DSL，似乎无法实现您想要的功能。但是，您可以编写一个自定义Transformer来重新实现提供所需语义的流表连接。

Kafka Streams - adding message frequency in enriched stream

1 个答案: