Question

我有一些类似以下的日志数据

日志

+---------------------+---------+---------+------+
|        date         | api_key | version | data |
+---------------------+---------+---------+------+
| 2018-05-08 01:00:00 | AAA     | v1      | data |
| 2018-05-08 02:00:00 | AAA     | v2      | data |
| 2018-05-06 03:00:00 | AAA     | v2      | data |
| 2018-05-06 04:00:00 | BBB     | v1      | data |
+---------------------+---------+---------+------+

date是API调用的日期
api_key是使用的API密钥
version是所用API的版本（我们有两个）

此数据目前存在于RDBMS中，我需要将数据移至Athena。

聚合

有一些遗留代码会定期运行以聚合日志。然后，此聚合数据将显示在仪表板上。从本质上讲，它将平面日志数据（上图）转换为由API密钥聚合的数据，根据版本和不同的时间窗口（所有时间，1天，7天等）提供调用的计数。

例如，以下聚合表显示聚合的原始数据。

+---------+------+---------+---------+--------+-----------+-----------+--------+-----------+-----------+
| api_key | hits | hits_v1 | hits_v2 | hits_1 | hits_1_v1 | hits_1_v2 | hits_7 | hits_7_v1 | hits_7_v2 |
+---------+------+---------+---------+--------+-----------+-----------+--------+-----------+-----------+
| AAA     |    3 |       1 |       2 |      2 |         1 |         1 |      3 |         1 |         2 |
| BBB     |    1 |       1 |       0 |      0 |         0 |         0 |      1 |         1 |         0 |
+---------+------+---------+---------+--------+-----------+-----------+--------+-----------+-----------+

这些是列的含义：

hits给定api-key
hits_v1给定api-key v1
hits_v2给定api-key v2
hits_1给定api-key
hits_1_v1给定v1

api-key

hits_1_v2给定v2

api-key

hits_7给定api-key
hits_7_v1给定v1

api-key

hits_7_v2给定v2

api-key

SQL

下面是我用来创建这个聚合表的SQL。

SELECT coalesce(hits_v1.api_key, hits_v2.api_key, hits_1_v1.api_key, hits_1_v2.api_key) api_key,
    coalesce(hits_v1.hits_v1,0) + coalesce(hits_v2.hits_v2,0) hits, 
    coalesce(hits_v1.hits_v1,0) hits_v1, 
    coalesce(hits_v2.hits_v2,0) hits_v2,

    coalesce(hits_1_v1.hits_1_v1,0) + coalesce(hits_1_v2.hits_1_v2,0) hits_1, 
    coalesce(hits_1_v1.hits_1_v1,0) hits_1_v1, 
    coalesce(hits_1_v2.hits_1_v2,0) hits_1_v2,

    coalesce(hits_7_v1.hits_7_v1,0) + coalesce(hits_7_v2.hits_7_v2,0) hits_7, 
    coalesce(hits_7_v1.hits_7_v1,0) hits_7_v1, 
    coalesce(hits_7_v2.hits_7_v2,0) hits_7_v2
FROM 
(
    (select api_key,count(*) as hits_v1 from logs where (version='v1' or version='') group by api_key) hits_v1
    FULL OUTER JOIN
    (select api_key,count(*) as hits_v2 from logs where version='v2' group by api_key) hits_v2                                                                       on hits_v2.api_key   = hits_v1.api_key

    FULL OUTER JOIN 
    (select api_key,count(*) as hits_1_v1 from logs where (version='v1' or version='') and (date > localtimestamp - interval '1' day) group by api_key) hits_1_v1    on hits_1_v1.api_key = coalesce(hits_v1.api_key, hits_v2.api_key)
    FULL OUTER JOIN 
    (select api_key,count(*) as hits_1_v2 from logs where version='v2' and (date > localtimestamp - interval '1' day) group by api_key) hits_1_v2                    on hits_1_v2.api_key = coalesce(hits_v1.api_key, hits_v2.api_key, hits_1_v1.api_key)

    FULL OUTER JOIN 
    (select api_key,count(*) as hits_7_v1 from logs where (version='v1' or version='') and (date > localtimestamp - interval '7' day) group by api_key) hits_7_v1    on hits_7_v1.api_key = coalesce(hits_v1.api_key, hits_v2.api_key, hits_1_v1.api_key, hits_1_v2.api_key)
    FULL OUTER JOIN 
    (select api_key,count(*) as hits_7_v2 from logs where version='v2' and (date > localtimestamp - interval '7' day) group by api_key) hits_7_v2                    on hits_7_v2.api_key = coalesce(hits_v1.api_key, hits_v2.api_key, hits_1_v1.api_key, hits_1_v2.api_key, hits_7_v1.api_key)
)
order by api_key asc

正如你所看到的，它非常重复，而且冗长。更糟糕的是，我需要添加的列越多（14天，30天，60天等），我每次都需要添加on子句以包含所有以前的连接。

这很有效，但我确信必须有更清洁的方法。有人可以帮忙吗？

PS。是的，我确实需要保留这个聚合表 - 它可能不太好，但是一大堆其他代码依赖于它，因此它无法改变。

Answer 1

使用条件聚合：

select api_key,
       sum(case when version = 'v1' or version = '' then 1 else 0 end) AS hits_v1,
       sum(case when version = 'v2' then 1 else 0 end) AS hits_v2,
       sum(case when (version = 'v1' or version = '') and (date > localtimestamp - interval '1' day) then 1 else 0 end) as hits_v1_1,
       . . .
from logs l
group by api_key;

Answer 2

您可以使用此查询代替子查询。

如果您想获得hits

hits给定api-key

您可以选择count(1)来获取所有数据。

SELECT api_key,
       count(1) hits,
       SUM(CASE WHEN (version='v1' or version='') THEN 1 ELSE 0 END ) hits_v1,
       SUM(CASE WHEN (version = 'v2' or version='') THEN 1 ELSE 0 END ) hits_v2,
       SUM(CASE WHEN (date > localtimestamp - interval '1' day) THEN 1 ELSE 0 END) hits_1,
       SUM(CASE WHEN (date > localtimestamp - interval '1' day) and (version='v1' or version='') THEN 1 ELSE 0 END) hits_1_v1,
       SUM(CASE WHEN (date > localtimestamp - interval '1' day) and (version='v2' or version='') THEN 1 ELSE 0 END) hits_1_v2,
       SUM(CASE WHEN (date > localtimestamp - interval '7' day) THEN 1 ELSE 0 END) hits_7,
       SUM(CASE WHEN  (version='v1' or version='') and (date > localtimestamp - interval '7' day) THEN 1 ELSE 0 END) hits_7_v1,
       SUM(CASE WHEN  (version='v2' or version='') and (date > localtimestamp - interval '7' day) THEN 1 ELSE 0 END) hits_7_v2
FROM logs
group by api_key

sqlfiddle：setRepeating()

多个完全外连接（来自同一表中的数据）

2 个答案: