这是关于postgres实际如何运作的更多理论问题。我有四个表,分别是A,B,C和D,分别有20k,870k,770k和1.5mln记录。 Ids是主键,并且正在外键上进行连接(默认情况下会对其进行索引,并且不允许空值)。这是我的疑问:
SELECT "A"."id" AS "a",
COUNT("B"."id") AS "b_count",
COUNT("C"."id") AS "c_count",
COUNT("D"."id") AS "d_count"
FROM "A"
LEFT OUTER JOIN "B" ON ("A"."id" = "B"."A_id")
LEFT OUTER JOIN "C" ON ("B"."id" = "C"."B_id")
LEFT OUTER JOIN "D" ON ("C"."id" = "D"."C_id")
GROUP BY "A"."id"
表和查询由django框架生成。由于不知情的原因,它运行了很长时间,服务器崩溃了。但是,如果我将这些LEFT OUTER JOIN中的任何一个更改为INNER JOIN,则查询将运行4.2秒。我知道LEFT和INNER JOIN在实践中是如何工作的,但为什么在这种情况下性能会有太大差异呢?
其他信息: 使用LEFT OUTER JOIN进行EXPLAIN查询(EXPLAIN(ANALYZE,BUFFERS)不起作用,因为查询没有结束):
"GroupAggregate (cost=606144.91..628195.81 rows=20784 width=16)"
" -> Sort (cost=606144.91..610513.52 rows=1747445 width=16)"
" Sort Key: A.id"
" -> Hash Right Join (cost=282896.74..365231.69 rows=1747445 width=16)"
" Hash Cond: (D.C_id = C.id)"
" -> Seq Scan on D (cost=0.00..32113.94 rows=1494094 width=8)"
" -> Hash (cost=267717.03..267717.03 rows=873257 width=12)"
" -> Hash Right Join (cost=208924.92..267717.03 rows=873257 width=12)"
" Hash Cond: (B.A_id = A.id)"
" -> Hash Right Join (cost=207935.28..250353.82 rows=873257 width=12)"
" Hash Cond: (C.B_id = B.id)"
" -> Seq Scan on C (cost=0.00..21039.49 rows=746649 width=8)"
" -> Hash (cost=193607.57..193607.57 rows=873257 width=8)"
" -> Seq Scan on B (cost=0.00..193607.57 rows=873257 width=8)"
" -> Hash (cost=729.84..729.84 rows=20784 width=4)"
" -> Seq Scan on A (cost=0.00..729.84 rows=20784 width=4)"
使用一个INNER JOIN而不是LEFT OUTER JOIN进行查询的EXPLAIN(ANALYZE,BUFFERS):
"GroupAggregate (cost=559935.67..578819.69 rows=20784 width=16) (actual time=4565.338..5090.632 rows=19567 loops=1)"
" Buffers: shared hit=10625 read=205521, temp read=27640 written=27388"
" -> Sort (cost=559935.67..563670.91 rows=1494094 width=16) (actual time=4565.244..4832.596 rows=1494094 loops=1)"
" Sort Key: A.id"
" Sort Method: external merge Disk: 37992kB"
" Buffers: shared hit=10625 read=205521, temp read=27640 written=27388"
" -> Hash Join (cost=278322.24..355638.06 rows=1494094 width=16) (actual time=2274.363..3341.921 rows=1494094 loops=1)"
" Hash Cond: (D.C_id = C.id)"
" Buffers: shared hit=10622 read=205521, temp read=13681 written=13429"
" -> Seq Scan on D (cost=0.00..32113.94 rows=1494094 width=8) (actual time=0.007..270.841 rows=1494094 loops=1)"
" Buffers: shared hit=3828 read=13345"
" -> Hash (cost=265343.13..265343.13 rows=746649 width=12) (actual time=2271.959..2271.959 rows=746649 loops=1)"
" Buckets: 4096 Batches: 64 Memory Usage: 512kB"
" Buffers: shared hit=6794 read=192176, temp read=5640 written=8351"
" -> Hash Join (cost=208924.92..265343.13 rows=746649 width=12) (actual time=1107.516..2138.249 rows=746649 loops=1)"
" Hash Cond: (B.A_id = A.id)"
" Buffers: shared hit=6794 read=192176, temp read=5640 written=5514"
" -> Hash Join (cost=207935.28..250353.82 rows=746649 width=12) (actual time=1099.799..1784.403 rows=746649 loops=1)"
" Hash Cond: (C.B_id = B.id)"
" Buffers: shared hit=6272 read=192176, temp read=5640 written=5514"
" -> Seq Scan on C (cost=0.00..21039.49 rows=746649 width=8) (actual time=0.005..203.923 rows=746649 loops=1)"
" Buffers: shared hit=4600 read=8973"
" -> Hash (cost=193607.57..193607.57 rows=873257 width=8) (actual time=1095.407..1095.407 rows=873453 loops=1)"
" Buckets: 4096 Batches: 64 Memory Usage: 547kB"
" Buffers: shared hit=1672 read=183203, temp written=2907"
" -> Seq Scan on B (cost=0.00..193607.57 rows=873257 width=8) (actual time=0.004..939.839 rows=873453 loops=1)"
" Buffers: shared hit=1672 read=183203"
" -> Hash (cost=729.84..729.84 rows=20784 width=4) (actual time=7.701..7.701 rows=20784 loops=1)"
" Buckets: 4096 Batches: 1 Memory Usage: 731kB"
" Buffers: shared hit=522"
" -> Seq Scan on A (cost=0.00..729.84 rows=20784 width=4) (actual time=0.004..4.661 rows=20784 loops=1)"
" Buffers: shared hit=522"
"Total runtime: 5099.046 ms"
奖励:如果我使用稍加修改的查询:
SELECT "A"."id" AS "a",
COUNT("B"."id") AS "b_count",
COUNT("C"."id") AS "c_count",
COUNT("D"."id") AS "d_count"
FROM "A"
LEFT OUTER JOIN "B" ON ("A"."id" = "B"."A_id")
LEFT OUTER JOIN "C" ON ("B"."id" = "C"."B_id")
LEFT OUTER JOIN "D" ON ("C"."id" = "D"."D_id")
WHERE "D"."id" < 1500000 -- this is always true
GROUP BY "A"."id"
然后它也可以工作,但比使用至少一个INNER JOIN要长一点。在这种情况下解析(分析,缓冲):
"GroupAggregate (cost=563670.91..582554.92 rows=20784 width=16) (actual time=6779.121..7286.640 rows=19567 loops=1)"
" Buffers: shared hit=10814 read=205329, temp read=27640 written=27388"
" -> Sort (cost=563670.91..567406.14 rows=1494094 width=16) (actual time=6777.606..7033.885 rows=1494094 loops=1)"
" Sort Key: A.id"
" Sort Method: external merge Disk: 37992kB"
" Buffers: shared hit=10814 read=205329, temp read=27640 written=27388"
" -> Hash Join (cost=278322.24..359373.29 rows=1494094 width=16) (actual time=4601.432..5674.321 rows=1494094 loops=1)"
" Hash Cond: (D.C_id = C.id)"
" Buffers: shared hit=10814 read=205329, temp read=13681 written=13429"
" -> Seq Scan on D (cost=0.00..35849.18 rows=1494094 width=8) (actual time=0.005..374.660 rows=1494094 loops=1)"
" Filter: (id < 1500000)"
" Buffers: shared hit=3892 read=13281"
" -> Hash (cost=265343.13..265343.13 rows=746649 width=12) (actual time=4600.782..4600.782 rows=746649 loops=1)"
" Buckets: 4096 Batches: 64 Memory Usage: 512kB"
" Buffers: shared hit=6922 read=192048, temp read=5640 written=8351"
" -> Hash Join (cost=208924.92..265343.13 rows=746649 width=12) (actual time=3363.352..4469.474 rows=746649 loops=1)"
" Hash Cond: (B.A_id = A.id)"
" Buffers: shared hit=6922 read=192048, temp read=5640 written=5514"
" -> Hash Join (cost=207935.28..250353.82 rows=746649 width=12) (actual time=3257.869..4066.067 rows=746649 loops=1)"
" Hash Cond: (C.B_id = B.id)"
" Buffers: shared hit=6400 read=192048, temp read=5640 written=5514"
" -> Seq Scan on C (cost=0.00..21039.49 rows=746649 width=8) (actual time=0.006..372.317 rows=746649 loops=1)"
" Buffers: shared hit=4664 read=8909"
" -> Hash (cost=193607.57..193607.57 rows=873257 width=8) (actual time=3257.327..3257.327 rows=873453 loops=1)"
" Buckets: 4096 Batches: 64 Memory Usage: 547kB"
" Buffers: shared hit=1736 read=183139, temp written=2907"
" -> Seq Scan on B (cost=0.00..193607.57 rows=873257 width=8) (actual time=0.004..3097.367 rows=873453 loops=1)"
" Buffers: shared hit=1736 read=183139"
" -> Hash (cost=729.84..729.84 rows=20784 width=4) (actual time=105.467..105.467 rows=20784 loops=1)"
" Buckets: 4096 Batches: 1 Memory Usage: 731kB"
" Buffers: shared hit=522"
" -> Seq Scan on A (cost=0.00..729.84 rows=20784 width=4) (actual time=0.002..101.506 rows=20784 loops=1)"
" Buffers: shared hit=522"
"Total runtime: 7294.388 ms"
为什么原始查询会破坏postgres,而其他两个就像魅力一样?
编辑: 即使没有count()子句,所有查询的行为都相同。问题出在仅JOIN中
答案 0 :(得分:0)
因为LEFT JOIN
可以将结果集乘以很多,具体取决于表格内容。
首先,您的查询似乎有误,您还会计算重复项,因此您应该使用COUNT(DISTINCT columnName)
SELECT "A"."id" AS "a",
COUNT(distinct "B"."id") AS "b_count",
COUNT(distinct "C"."id") AS "c_count",
COUNT(distinct "D"."id") AS "d_count"
FROM "A"
LEFT OUTER JOIN "B" ON ("A"."id" = "B"."A_id")
LEFT OUTER JOIN "C" ON ("B"."id" = "C"."B_id")
LEFT OUTER JOIN "D" ON ("C"."id" = "D"."C_id")
GROUP BY "A"."id
其次,考虑添加适当的索引。
A - (id)
B - (id,A_id)
C - (id,B_id)
D - (id,C_id)