Question

说我有一个有关温度预测数据的主题，如下所示：

2018-10-25,Melbourne,21
2018-10-26,Melbourne,17
2018-10-27,Melbourne,21
2018-10-25,Sydney,22
2018-10-26,Sydney,20
2018-10-27,Sydney,23
2018-10-26,Melbourne,18
2018-10-27,Melbourne,22
2018-10-26,Sydney,21
2018-10-27,Sydney,24

每个条目均包含日期，城市和预报温度，并代表该日期对该城市的预报的更新。我可以将其描述为这样的KSQL流：

CREATE STREAM forecasts_csv ( \
  date VARCHAR, \
  city VARCHAR, \
  temperature INTEGER \
) WITH (kafka_topic='forecasts-csv', value_format='DELIMITED');

现在，我想要一个表，该表代表每个城市的当前（即最新）预测温度，以及该城市随时间推移的最低和最高温度。所需的示例输出是：

{ date='2018-10-27', city='Melbourne', latest=22, min=21, max=22 }

我该如何实现？

我设法按如下方式获得总量（最小/最大）：

CREATE STREAM forecasts_keyed \
WITH (partitions=4, value_format='JSON') \
AS SELECT date + '/' + city AS forecast_key, * \
FROM forecasts_csv \
PARTITION BY forecast_key;

CREATE TABLE forecasts_minmax \
WITH (partitions=4, value_format='JSON') \
AS SELECT forecast_key, date, city, \
          min(temperature) as min, max(temperature) as max \
FROM forecasts_keyed \
GROUP by forecast_key, date, city;

这给了我输出消息，例如：

{"FORECAST_KEY":"2018-10-27/Melbourne","DATE":"2018-10-27","CITY":"Melbourne","MIN":21,"MAX":22}

但是我不知道如何将其与“最新”阅读结合起来。

Answer 1

您需要实现一个UDAF，我们将其称为LATEST，它可以保留给定列和键的最新值。这非常简单，您可以在KSQL文档中找到如何添加自定义UDAF：https://docs.confluent.io/current/ksql/docs/developer-guide/udf.html#udafs

假设您拥有LATEST UDAF，则可以编写以下查询：

CREATE TABLE foo AS
SELECT
  date,
  city,
  MIN(temperature) AS minValue,
  MAX(temperature) AS maxValue,
  LATEST(temperature) AS latestValue
FROM forecasts_csv
GROUP BY date, city;

如何使用复合键从主题创建KSQL表？

1 个答案: