如何获取Kafka流聚合任务的结果并将数据发送到另一个服务?

时间:2018-08-22 03:35:48

标签: apache-kafka apache-kafka-streams

我使用Kafka流来处理实时数据,并且需要对窗口时间数据进行一些汇总操作。

关于聚合操作,我有两个问题。

  1. 如何获取汇总数据?我需要将其发送到第三服务。
  2. 聚合操作后,我无法向第三服务发送消息,代码也未运行。

这是我的代码:

stream = builder.stream("topic");
windowedKStream = stream.map(XXXXX).groupByKey().windowedBy("5mins");
ktable = windowedKStream.aggregate(()->"", new Aggregator(K,V,result));

// my data is stored in 'result' variable, but I can't get it at the end of the 5 mins window. 
// I need to send the 'result' to a 3rd service. But I don't know where to temporarily store it and then how to get it.

// below is the code the call a 3rd service, but the code can't be executed(reachable).
// I think it should be executed every 5 mins when thewindows is over. But it isn't.

result = httpclient.execute('result');

1 个答案:

答案 0 :(得分:2)

我想可能想做些类似的事情:

ktable.toStream().foreach((k,v) -> httpclient.execute(v));

每次KTable被更新(禁用缓存)时​​,更新记录将被发送到下游,并且foreach将以v作为当前聚合结果被执行。