我发现自己越来越频繁地做这样的统计 - 对于tbl1中的每个记录，使用它作为扫描tbl2的条件并在tbl2上进行聚合，返回聚合结果。使用子查询很容易。
但我需要在同一子查询中一次传递多次聚合，而不是多次扫描。
例如，以下是一些分析样本数据：
SELECT to_timestamp((random()*999999)) evt_ts,
evt_id
INTO temp.test_so_44410354
FROM
(
SELECT *, (random()*2000)::int AS idx
FROM generate_series( 1, 1000) AS evt_id
ORDER BY idx
) AS t
evt_ts |evt_id |
--------------------|-------|
1970-01-01 08:13:15 |109 |
1970-01-01 08:17:22 |762 |
1970-01-01 08:19:28 |630 |
1970-01-01 08:29:34 |429 |
1970-01-01 08:48:28 |70 |
要查看事件在给定时间范围内的分布情况， 目前的方法和查询计划＆amp;期望的输出：
With wcal AS (
SELECT ts, ts::date as dt
FROM generate_series( '1970-01-01 0:0:0', '1970-01-12 0:0:0', INTERVAL '1 day' ) AS ts
)
SELECT ts, dt
,(
SELECT count(evt_ts)
FROM temp.test_so_44410354 as el
WHERE ( el.evt_ts < wcal.ts AND el.evt_ts > wcal.ts - INTERVAL '1 day' )
) as ecnt
,(
SELECT sum(evt_id) -- should be array_agg, sum is locate to the difference in the results.
FROM temp.test_so_44410354 as el
WHERE ( el.evt_ts < wcal.ts AND el.evt_ts > wcal.ts - INTERVAL '1 day' )
) as eids
FROM wcal
QUERY PLAN |
----------------------------------------------------------------------------------------------------|
CTE Scan on wcal (cost=12.50..21314.08 rows=1000 width=28) |
CTE wcal |
-> Function Scan on generate_series ts (cost=0.00..12.50 rows=1000 width=12) |
SubPlan 2 |
-> Aggregate (cost=10.63..10.64 rows=1 width=8) |
-> Bitmap Heap Scan on test_so_44410354 el (cost=4.33..10.62 rows=5 width=8) |
Recheck Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval))) |
-> Bitmap Index Scan on test_so_44410354_idx (cost=0.00..4.33 rows=5 width=0) |
Index Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval))) |
SubPlan 3 |
-> Aggregate (cost=10.63..10.64 rows=1 width=8) |
...
ts |dt |ecnt |eids |
--------------------|-----------|-----|------|
1970-01-01 00:00:00 |1970-01-01 |0 | |
1970-01-02 00:00:00 |1970-01-02 |68 |36156 |
1970-01-03 00:00:00 |1970-01-03 |85 |42103 |
1970-01-04 00:00:00 |1970-01-04 |94 |47092 |
1970-01-05 00:00:00 |1970-01-05 |74 |36276 |
....
有2个子计划，似乎 pg必须扫描表evt_log两次才能进行两次相同的聚合。我想避免这个。因为聚合条件完全相同，如果它可以在一次通过中做很多统计聚合，它可能会快得多。日志可能会变得非常大，处理将花费相当长的时间。
对拟议解决方案的反馈I.
嵌套join未按预期工作：
With wcal AS (
SELECT ts, ts::date as dt
FROM generate_series( '1970-01-01 0:0:0', '1970-01-12 0:0:0', INTERVAL '1 day' ) AS ts
)
, ag AS (
SELECT
count(evt_ts) AS ecnt
, sum(evt_id) AS eids
FROM temp.test_so_44410354 as el
JOIN wcal ON ( el.evt_ts < wcal.ts AND el.evt_ts > wcal.ts - INTERVAL '1 day' )
)
SELECT ts, dt, ag.ecnt, eids
FROM wcal
JOIN ag ON true
QUERY PLAN |
----------------------------------------------------------------------------------------------------------------------|
Nested Loop (cost=4239.57..4269.59 rows=1000 width=28) |
CTE wcal |
-> Function Scan on generate_series ts (cost=0.00..12.50 rows=1000 width=12) |
CTE ag |
-> Aggregate (cost=4227.06..4227.07 rows=1 width=16) |
-> Nested Loop (cost=0.28..3671.50 rows=111111 width=12) |
-> CTE Scan on wcal wcal_1 (cost=0.00..20.00 rows=1000 width=8) |
-> Index Scan using test_so_44410354_idx on test_so_44410354 el (cost=0.28..2.54 rows=111 width=12) |
Index Cond: ((evt_ts < wcal_1.ts) AND (evt_ts > (wcal_1.ts - '1 day'::interval))) |
-> CTE Scan on ag (cost=0.00..0.02 rows=1 width=16) |
-> CTE Scan on wcal (cost=0.00..20.00 rows=1000 width=12) |
ts |dt |ecnt |eids |
--------------------|-----------|-----|-------|
1970-01-01 00:00:00 |1970-01-01 |920 |464436 |
1970-01-04 00:00:00 |1970-01-04 |920 |464436 |
1970-01-05 00:00:00 |1970-01-05 |920 |464436 |
1970-01-06 00:00:00 |1970-01-06 |920 |464436 |
1970-01-07 00:00:00 |1970-01-07 |920 |464436 |
....
拟议解决方案II的反馈： 写作不同但查询计划与2子计划基本相同。
DROP TYPE temp.my_agg ;
CREATE TYPE temp.my_agg AS ( ecnt int, eids int );
EXPLAIN
With wcal AS (
SELECT ts, ts::date as dt
FROM generate_series( '1970-01-01 0:0:0', '1970-01-12 0:0:0', INTERVAL '1 day' ) AS ts
)
SELECT ts, dt
,(
SELECT (count(evt_ts),sum(evt_id))::temp.my_agg
FROM temp.test_so_44410354 as el
WHERE ( el.evt_ts < wcal.ts AND el.evt_ts > wcal.ts - INTERVAL '1 day' )
).*
FROM wcal
QUERY PLAN |
--------------------------------------------------------------------------------------------------------------------------------------------|
CTE Scan on wcal (cost=12.50..21349.08 rows=1000 width=20) (actual time=0.182..3.518 rows=12 loops=1) |
CTE wcal |
-> Function Scan on generate_series ts (cost=0.00..12.50 rows=1000 width=12) (actual time=0.057..0.083 rows=12 loops=1) |
SubPlan 2 |
-> Aggregate (cost=10.64..10.66 rows=1 width=32) (actual time=0.140..0.140 rows=1 loops=12) |
-> Bitmap Heap Scan on test_so_44410354 el (cost=4.33..10.62 rows=5 width=12) (actual time=0.049..0.083 rows=77 loops=12) |
Recheck Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval))) |
Heap Blocks: exact=66 |
-> Bitmap Index Scan on test_so_44410354_idx (cost=0.00..4.33 rows=5 width=0) (actual time=0.037..0.037 rows=77 loops=12) |
Index Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval))) |
SubPlan 3 |
...
Planning time: 0.548 ms |
Execution time: 3.748 ms |
对于建议的解决方案II，如果聚合作为记录返回，则查询计划如下所示： 这大约加倍了速度c.f.以前的查询。 （但最终结果不需要记录类型元素。）
QUERY PLAN |
--------------------------------------------------------------------------------------------------------------------------------------------|
CTE Scan on wcal (cost=12.50..10690.79 rows=1000 width=44) (actual time=0.147..1.979 rows=12 loops=1) |
CTE wcal |
-> Function Scan on generate_series ts (cost=0.00..12.50 rows=1000 width=12) (actual time=0.057..0.082 rows=12 loops=1) |
SubPlan 2 |
-> Aggregate (cost=10.64..10.66 rows=1 width=32) (actual time=0.151..0.151 rows=1 loops=12) |
-> Bitmap Heap Scan on test_so_44410354 el (cost=4.33..10.62 rows=5 width=12) (actual time=0.054..0.089 rows=77 loops=12) |
Recheck Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval))) |
Heap Blocks: exact=66 |
-> Bitmap Index Scan on test_so_44410354_idx (cost=0.00..4.33 rows=5 width=0) (actual time=0.040..0.040 rows=77 loops=12) |
Index Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval))) |
Planning time: 0.381 ms |
Execution time: 2.147 ms |
LATERAL加入解决方案的工作方式与预期一致。感谢。
EXPLAIN ANALYZE
With wcal AS (
SELECT ts, ts::date as dt
FROM generate_series( '1970-01-01 0:0:0', '1970-01-12 0:0:0', INTERVAL '1 day' ) AS ts
)
SELECT ts, dt, p.*
FROM wcal
CROSS JOIN LATERAL (
SELECT count(evt_ts), sum(evt_id)
FROM temp.test_so_44410354 as el
WHERE ( el.evt_ts < wcal.ts AND el.evt_ts > wcal.ts - INTERVAL '1 day' )
) p
QUERY PLAN |
------------------------------------------------------------------------------------------------------------------------------------------|
Nested Loop (cost=23.15..10705.79 rows=1000 width=28) (actual time=0.130..1.611 rows=12 loops=1) |
CTE wcal |
-> Function Scan on generate_series ts (cost=0.00..12.50 rows=1000 width=12) (actual time=0.054..0.075 rows=12 loops=1) |
-> CTE Scan on wcal (cost=0.00..20.00 rows=1000 width=12) (actual time=0.061..0.099 rows=12 loops=1) |
-> Aggregate (cost=10.64..10.65 rows=1 width=16) (actual time=0.123..0.123 rows=1 loops=12) |
-> Bitmap Heap Scan on test_so_44410354 el (cost=4.33..10.62 rows=5 width=12) (actual time=0.044..0.074 rows=77 loops=12) |
Recheck Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval))) |
Heap Blocks: exact=66 |
-> Bitmap Index Scan on test_so_44410354_idx (cost=0.00..4.33 rows=5 width=0) (actual time=0.033..0.033 rows=77 loops=12) |
Index Cond: ((evt_ts < wcal.ts) AND (evt_ts > (wcal.ts - '1 day'::interval))) |
Planning time: 0.399 ms |
Execution time: 1.778 ms |
ts |dt |count |sum |
--------------------|-----------|------|------|
1970-01-01 00:00:00 |1970-01-01 |0 | |
1970-01-02 00:00:00 |1970-01-02 |68 |36156 |
1970-01-03 00:00:00 |1970-01-03 |85 |42103 |
1970-01-04 00:00:00 |1970-01-04 |94 |47092 |
1970-01-05 00:00:00 |1970-01-05 |74 |36276 |
1970-01-06 00:00:00 |1970-01-06 |96 |43797 |
答案 0 :(得分：2)
With wcal AS (
SELECT ts, ts::date as dt,
extract( week from ts) as wk,
extract( month from ts ) as mo,
extract( quater from ts ) as qtr,
FROM generate_series( t1, t2, tspan )
)
, ag as (
SELECT count(evt_ts) ecnt, array_aggr(evt_id) eids
FROM ev_log as el
JOIN wcal on (el.ts < wcal.ts AND el.ts < wcal.ts - tspan)
)
SELECT ts, dt, wk, mo, qtr, ecnt, eids
FROM wcal
JOIN ag on true
我不确定连接条件 - 您需要将整个查询放到OP才能猜到
已更新
根据Abelisto通知 - 谢谢
答案 1 :(得分：1)
有点棘手，但是：
create type t_my_aggs as (ecnt bigint, eids bigint[]);
With wcal AS (
SELECT ts, ts::date as dt,
extract( week from ts) as wk,
extract( month from ts ) as mo,
extract( quater from ts ) as qtr,
FROM generate_series( t1, t2, tspan )
)
SELECT ts, dt, wk, mo, qtr
,(
SELECT row(count(evt_ts), array_agg(evt_id))::t_my_aggs
FROM ev_log as el
WHERE ( el.ts < wcal.ts AND el.ts < wcal.ts - tspan )
).* from wcal;
答案 2 :(得分：0)
使用lateral join：
With wcal AS (
SELECT ts, ts::date as dt,
extract( week from ts) as wk,
extract( month from ts ) as mo,
extract( quater from ts ) as qtr,
FROM generate_series( t1, t2, tspan )
)
SELECT ts, dt, wk, mo, qtr, ecnt, eids
from
wcal cross join lateral (
SELECT count(evt_ts) as ecnt, array_agg(evt_id) as eids
FROM ev_log as el
WHERE (el.ts < wcal.ts AND el.ts < wcal.ts - tspan)) as el;
<table> cross join lateral (<query>)是<table>, lateral (<query>)的同义词。