Postgresql - LEFT OUTER JOIN性能问题

时间:2016-06-07 10:09:14

标签: performance postgresql left-join

这是关于postgres实际如何运作的更多理论问题。我有四个表,分别是A,B,C和D,分别有20k,870k,770k和1.5mln记录。 Ids是主键,并且正在外键上进行连接(默认情况下会对其进行索引,并且不允许空值)。这是我的疑问:

SELECT "A"."id" AS "a", 
  COUNT("B"."id") AS "b_count", 
   COUNT("C"."id") AS "c_count",
   COUNT("D"."id") AS "d_count" 
FROM "A" 
LEFT OUTER JOIN "B" ON ("A"."id" = "B"."A_id") 
LEFT OUTER JOIN "C" ON ("B"."id" = "C"."B_id") 
LEFT OUTER JOIN "D" ON ("C"."id" = "D"."C_id") 
GROUP BY "A"."id"

表和查询由django框架生成。由于不知情的原因,它运行了很长时间,服务器崩溃了。但是,如果我将这些LEFT OUTER JOIN中的任何一个更改为INNER JOIN,则查询将运行4.2秒。我知道LEFT和INNER JOIN在实践中是如何工作的,但为什么在这种情况下性能会有太大差异呢?

其他信息: 使用LEFT OUTER JOIN进行EXPLAIN查询(EXPLAIN(ANALYZE,BUFFERS)不起作用,因为查询没有结束):

"GroupAggregate  (cost=606144.91..628195.81 rows=20784 width=16)"
"  ->  Sort  (cost=606144.91..610513.52 rows=1747445 width=16)"
"        Sort Key: A.id"
"        ->  Hash Right Join  (cost=282896.74..365231.69 rows=1747445 width=16)"
"              Hash Cond: (D.C_id = C.id)"
"              ->  Seq Scan on D  (cost=0.00..32113.94 rows=1494094 width=8)"
"              ->  Hash  (cost=267717.03..267717.03 rows=873257 width=12)"
"                    ->  Hash Right Join  (cost=208924.92..267717.03 rows=873257 width=12)"
"                          Hash Cond: (B.A_id = A.id)"
"                          ->  Hash Right Join  (cost=207935.28..250353.82 rows=873257 width=12)"
"                                Hash Cond: (C.B_id = B.id)"
"                                ->  Seq Scan on C  (cost=0.00..21039.49 rows=746649 width=8)"
"                                ->  Hash  (cost=193607.57..193607.57 rows=873257 width=8)"
"                                      ->  Seq Scan on B  (cost=0.00..193607.57 rows=873257 width=8)"
"                          ->  Hash  (cost=729.84..729.84 rows=20784 width=4)"
"                                ->  Seq Scan on A  (cost=0.00..729.84 rows=20784 width=4)"

使用一个INNER JOIN而不是LEFT OUTER JOIN进行查询的EXPLAIN(ANALYZE,BUFFERS):

"GroupAggregate  (cost=559935.67..578819.69 rows=20784 width=16) (actual time=4565.338..5090.632 rows=19567 loops=1)"
"  Buffers: shared hit=10625 read=205521, temp read=27640 written=27388"
"  ->  Sort  (cost=559935.67..563670.91 rows=1494094 width=16) (actual time=4565.244..4832.596 rows=1494094 loops=1)"
"        Sort Key: A.id"
"        Sort Method: external merge  Disk: 37992kB"
"        Buffers: shared hit=10625 read=205521, temp read=27640 written=27388"
"        ->  Hash Join  (cost=278322.24..355638.06 rows=1494094 width=16) (actual time=2274.363..3341.921 rows=1494094 loops=1)"
"              Hash Cond: (D.C_id = C.id)"
"              Buffers: shared hit=10622 read=205521, temp read=13681 written=13429"
"              ->  Seq Scan on D  (cost=0.00..32113.94 rows=1494094 width=8) (actual time=0.007..270.841 rows=1494094 loops=1)"
"                    Buffers: shared hit=3828 read=13345"
"              ->  Hash  (cost=265343.13..265343.13 rows=746649 width=12) (actual time=2271.959..2271.959 rows=746649 loops=1)"
"                    Buckets: 4096  Batches: 64  Memory Usage: 512kB"
"                    Buffers: shared hit=6794 read=192176, temp read=5640 written=8351"
"                    ->  Hash Join  (cost=208924.92..265343.13 rows=746649 width=12) (actual time=1107.516..2138.249 rows=746649 loops=1)"
"                          Hash Cond: (B.A_id = A.id)"
"                          Buffers: shared hit=6794 read=192176, temp read=5640 written=5514"
"                          ->  Hash Join  (cost=207935.28..250353.82 rows=746649 width=12) (actual time=1099.799..1784.403 rows=746649 loops=1)"
"                                Hash Cond: (C.B_id = B.id)"
"                                Buffers: shared hit=6272 read=192176, temp read=5640 written=5514"
"                                ->  Seq Scan on C  (cost=0.00..21039.49 rows=746649 width=8) (actual time=0.005..203.923 rows=746649 loops=1)"
"                                      Buffers: shared hit=4600 read=8973"
"                                ->  Hash  (cost=193607.57..193607.57 rows=873257 width=8) (actual time=1095.407..1095.407 rows=873453 loops=1)"
"                                      Buckets: 4096  Batches: 64  Memory Usage: 547kB"
"                                      Buffers: shared hit=1672 read=183203, temp written=2907"
"                                      ->  Seq Scan on B  (cost=0.00..193607.57 rows=873257 width=8) (actual time=0.004..939.839 rows=873453 loops=1)"
"                                            Buffers: shared hit=1672 read=183203"
"                          ->  Hash  (cost=729.84..729.84 rows=20784 width=4) (actual time=7.701..7.701 rows=20784 loops=1)"
"                                Buckets: 4096  Batches: 1  Memory Usage: 731kB"
"                                Buffers: shared hit=522"
"                                ->  Seq Scan on A  (cost=0.00..729.84 rows=20784 width=4) (actual time=0.004..4.661 rows=20784 loops=1)"
"                                      Buffers: shared hit=522"
"Total runtime: 5099.046 ms"

奖励:如果我使用稍加修改的查询:

SELECT "A"."id" AS "a", 
  COUNT("B"."id") AS "b_count", 
   COUNT("C"."id") AS "c_count",
   COUNT("D"."id") AS "d_count" 
FROM "A" 
LEFT OUTER JOIN "B" ON ("A"."id" = "B"."A_id") 
LEFT OUTER JOIN "C" ON ("B"."id" = "C"."B_id") 
LEFT OUTER JOIN "D" ON ("C"."id" = "D"."D_id") 
WHERE "D"."id" < 1500000 -- this is always true
GROUP BY "A"."id"

然后它也可以工作,但比使用至少一个INNER JOIN要长一点。在这种情况下解析(分析,缓冲):

"GroupAggregate  (cost=563670.91..582554.92 rows=20784 width=16) (actual time=6779.121..7286.640 rows=19567 loops=1)"
"  Buffers: shared hit=10814 read=205329, temp read=27640 written=27388"
"  ->  Sort  (cost=563670.91..567406.14 rows=1494094 width=16) (actual time=6777.606..7033.885 rows=1494094 loops=1)"
"        Sort Key: A.id"
"        Sort Method: external merge  Disk: 37992kB"
"        Buffers: shared hit=10814 read=205329, temp read=27640 written=27388"
"        ->  Hash Join  (cost=278322.24..359373.29 rows=1494094 width=16) (actual time=4601.432..5674.321 rows=1494094 loops=1)"
"              Hash Cond: (D.C_id = C.id)"
"              Buffers: shared hit=10814 read=205329, temp read=13681 written=13429"
"              ->  Seq Scan on D  (cost=0.00..35849.18 rows=1494094 width=8) (actual time=0.005..374.660 rows=1494094 loops=1)"
"                    Filter: (id < 1500000)"
"                    Buffers: shared hit=3892 read=13281"
"              ->  Hash  (cost=265343.13..265343.13 rows=746649 width=12) (actual time=4600.782..4600.782 rows=746649 loops=1)"
"                    Buckets: 4096  Batches: 64  Memory Usage: 512kB"
"                    Buffers: shared hit=6922 read=192048, temp read=5640 written=8351"
"                    ->  Hash Join  (cost=208924.92..265343.13 rows=746649 width=12) (actual time=3363.352..4469.474 rows=746649 loops=1)"
"                          Hash Cond: (B.A_id = A.id)"
"                          Buffers: shared hit=6922 read=192048, temp read=5640 written=5514"
"                          ->  Hash Join  (cost=207935.28..250353.82 rows=746649 width=12) (actual time=3257.869..4066.067 rows=746649 loops=1)"
"                                Hash Cond: (C.B_id = B.id)"
"                                Buffers: shared hit=6400 read=192048, temp read=5640 written=5514"
"                                ->  Seq Scan on C  (cost=0.00..21039.49 rows=746649 width=8) (actual time=0.006..372.317 rows=746649 loops=1)"
"                                      Buffers: shared hit=4664 read=8909"
"                                ->  Hash  (cost=193607.57..193607.57 rows=873257 width=8) (actual time=3257.327..3257.327 rows=873453 loops=1)"
"                                      Buckets: 4096  Batches: 64  Memory Usage: 547kB"
"                                      Buffers: shared hit=1736 read=183139, temp written=2907"
"                                      ->  Seq Scan on B  (cost=0.00..193607.57 rows=873257 width=8) (actual time=0.004..3097.367 rows=873453 loops=1)"
"                                            Buffers: shared hit=1736 read=183139"
"                          ->  Hash  (cost=729.84..729.84 rows=20784 width=4) (actual time=105.467..105.467 rows=20784 loops=1)"
"                                Buckets: 4096  Batches: 1  Memory Usage: 731kB"
"                                Buffers: shared hit=522"
"                                ->  Seq Scan on A  (cost=0.00..729.84 rows=20784 width=4) (actual time=0.002..101.506 rows=20784 loops=1)"
"                                      Buffers: shared hit=522"
"Total runtime: 7294.388 ms"

为什么原始查询会破坏postgres,而其他两个就像魅力一样?

编辑: 即使没有count()子句,所有查询的行为都相同。问题出在仅JOIN中

1 个答案:

答案 0 :(得分:0)

因为LEFT JOIN可以将结果集乘以很多,具体取决于表格内容。

首先,您的查询似乎有误,您还会计算重复项,因此您应该使用COUNT(DISTINCT columnName)

SELECT "A"."id" AS "a", 
  COUNT(distinct "B"."id") AS "b_count", 
   COUNT(distinct "C"."id") AS "c_count",
   COUNT(distinct "D"."id") AS "d_count" 
FROM "A" 
LEFT OUTER JOIN "B" ON ("A"."id" = "B"."A_id") 
LEFT OUTER JOIN "C" ON ("B"."id" = "C"."B_id") 
LEFT OUTER JOIN "D" ON ("C"."id" = "D"."C_id") 
GROUP BY "A"."id

其次,考虑添加适当的索引。

A - (id)
B - (id,A_id)
C - (id,B_id)
D - (id,C_id)