索引列的Postgresql ORDER BY很慢

时间:2014-06-19 13:31:55

标签: sql postgresql

我提出了一个非常大的问题,最终我要按照一些标准进行排序。查询的主表有大约1 500 000行,并且我有很多JOIN语句。当我运行查询时,它需要超过9分钟,当运行EXPLAIN SELECT时,很明显它是" ORDER BY"条款,这需要很多时间。 ORDER BY语句之后的列使用btree编制索引,并且是BIGINT。任何人都可以建议我可以做些什么来缩短执行时间?以下是EXPLAIN ANALYZE SELECT ...的一部分(完成时间超过9分钟)

查询:

explain ANALYSE
select aa2.aid as cid
,aa2.id ,aa2.ty 
,coalesce(sign.bb,'') as tyy
,aa2.rd ,aa2.dt ,aa2.appliedon 
,aa2.approvedon ,aa2.pprec 
,aa2.ppret ,aa2.cdate ,pm.pmd,tblapplic.dob 
,coalesce(tblapplic.firstname ,'') as fname 
,coalesce(tblapplic.middlename ,'') as mname 
,coalesce(tblapplic.surname ,'') as sname 
,coalesce(Upper(dcou.descr),'') as dcouu 
,coalesce(aa2.visaid,0) as visaid 
,coalesce(tblcountry.descr,'') as visaname 
,(SELECT chr(9) || array_to_string(ARRAY(SELECT descr FROM tblcountry WHERE dcountry=travdets.dcountry ORDER BY descr),chr(9))) as visanamecombo 
,coalesce(aa2.stype,0) as stypeid ,coalesce(stype.descr,'') as stype 
,coalesce(aa2.sts,0) as appStatusid ,coalesce(sta.descr,'') as appStatus 
,lofficesuperuser.descr as lofficesuperuser,coalesce(aa2.loffice,0) as lofficeid 
,coalesce(lofficesuperuser.descr,coalesce(loffice.descr,'')) as loffice 
,coalesce(pm.id,0) as pmid 
,(SELECT chr(9) || array_to_string(ARRAY(SELECT descr FROM stype WHERE typee=aa2.ty and visaa=coalesce(aa2.visaid,0) ORDER BY descr),chr(9))) as sTypeCombo 
,(SELECT chr(9) || array_to_string(ARRAY(SELECT descr FROM sta WHERE typee=aa2.ty ORDER BY descr),chr(9))) as appStatusCombo 
,coalesce(aa2.colltype,0) as colltypeid
,coalesce(tblcolltype.descr,'') as colltype,aa2.embcolldt as embcolldt
,aa2.clcolldt as clbcolldt,travdets.leavingdt as leaving,coalesce(aa2.pptrnum,'') as ppTrackNumber
,coalesce(aa2.rt,0) as regtypeid,coalesce(tbllist1.descr,'') as regtype
,coalesce(aa2.reff,0) as referralid ,coalesce(tblreferral.descr,'') as referral
,coalesce(aa2.scaller,0) as callerid 
,coalesce(namess.descr,'') as caller,coalesce(aa2.travdets,0) as travdetsid
,coalesce(curact.luser,-1) as actionluser
,aa2.iss 
from aa2  
LEFT JOIN tblapplic on aa2.aid=tblapplic.idnumber  
left JOIN travdets on aa2.travdets=travdets.idnumber  
left JOIN pm on aa2.id=pm.aa2id  and pm.pagee=aa2.ty 
LEFT JOIN sign ON aa2.ty = sign.aa  
LEFT JOIN tblnationality dcou ON dcou.idnumber=travdets.dcountry 
left join tblcountry on aa2.visaid=tblcountry.idnumber  
left join aa2pre on aa2.id=aa2pre.aa2id 
left join tbluser on aa2pre.addusr=tbluser.idnumber  
left join loffice on tbluser.office=loffice.idnumber  
left join loffice lofficesuperuser on aa2.loffice=lofficesuperuser.idnumber  
left outer join stype on aa2.stype=stype.id  
left outer join sta on aa2.sts=sta.idnumber and sta.typee=aa2.ty 
left join tblcolltype on aa2.colltype=tblcolltype.idnumber  
left join tbllist1 on aa2.rt=tbllist1.idnumber  
left join tblreferral on aa2.reff=tblreferral.idnumber  
left join namess on aa2.scaller=namess.idnumber  
left join aa2_curr_act curact on aa2.id=curact.aa2id 
where aa2.op_status=0 and aa2.ty>0
ORDER BY aa2.aid DESC, aa2.ty DESC 
LIMIT 1000

计划:

Limit  (cost=2502213.29..2502215.79 rows=1000 width=555) (actual time=569132.700..569133.021 rows=1000 loops=1)
  ->  Sort  (cost=2502213.29..2502284.78 rows=28593 width=555) (actual time=569132.699..569132.870 rows=1000 loops=1)
    Sort Key: aa2.aid, aa2.ty
    Sort Method: top-N heapsort  Memory: 3539kB
    ->  Hash Left Join  (cost=424537.96..2500645.57 rows=28593 width=555) (actual time=3817.372..565313.067 rows=1175709 loops=1)
          Hash Cond: (aa2.id = curact.aa2id)
          ->  Hash Left Join  (cost=424498.48..469748.41 rows=28593 width=547) (actual time=3816.435..28001.006 rows=1175709 loops=1)
                Hash Cond: (aa2.scaller = namess.idnumber)
                ->  Hash Left Join  (cost=424492.25..469350.45 rows=28593 width=532) (actual time=3816.326..27006.364 rows=1175709 loops=1)
                      Hash Cond: (aa2.reff = tblreferral.idnumber)
                      ->  Hash Left Join  (cost=424452.85..468924.32 rows=28593 width=506) (actual time=3815.846..25976.031 rows=1175709 loops=1)
                            Hash Cond: (aa2.rt = tbllist1.idnumber)
                            ->  Nested Loop Left Join  (cost=424451.44..468537.08 rows=28593 width=487) (actual time=3815.819..25098.987 rows=1175709 loops=1)
                                  Join Filter: (aa2.colltype = tblcolltype.idnumber)"
                                  Rows Removed by Join Filter: 8229962
                                  ->  Hash Left Join  (cost=424451.44..465533.73 rows=28593 width=451) (actual time=3815.793..21647.918 rows=1175709 loops=1)
                                        Hash Cond: ((aa2.sts = public.sta.idnumber) AND (aa2.ty = public.sta.typee))
                                        ->  Hash Left Join  (cost=424443.94..464745.17 rows=28593 width=424) (actual time=3815.677..20748.314 rows=1175709 loops=1)
                                              Hash Cond: (aa2.stype = public.stype.id)"
                                              ->  Hash Left Join  (cost=424367.49..464239.83 rows=28593 width=383) (actual time=3814.669..19892.991 rows=1175709 loops=1)
                                                    Hash Cond: (aa2.loffice = lofficesuperuser.idnumber)
                                                    ->  Hash Left Join  (cost=424366.18..464131.28 rows=28593 width=374) (actual time=3814.651..19300.840 rows=1175709 loops=1)
                                                          Hash Cond: (tbluser.office = loffice.idnumber)
                                                          ->  Hash Left Join  (cost=424364.86..464002.72 rows=28593 width=373) (actual time=3814.628..18721.215 rows=1175709 loops=1)
                                                                Hash Cond: (aa2pre.addusr = tbluser.idnumber)
                                                                ->  Hash Left Join  (cost=424329.53..463574.24 rows=28593 width=373) (actual time=3813.991..18128.257 rows=1175709 loops=1)
                                                                      Hash Cond: (aa2.id = aa2pre.aa2id)
                                                                      ->  Hash Left Join  (cost=424309.57..463447.04 rows=28593 width=365) (actual time=3813.404..17427.883 rows=1175708 loops=1)
                                                                            Hash Cond: (aa2.visaid = public.tblcountry.idnumber)
                                                                            ->  Hash Left Join  (cost=424286.05..463120.22 rows=28593 width=342) (actual time=3813.207..16687.672 rows=1175708 loops=1)
                                                                                  Hash Cond: (travdets.dcountry = dcou.idnumber)
                                                                                  ->  Hash Left Join  (cost=424277.35..462718.39 rows=28593 width=330) (actual time=3813.056..15885.221 rows=1175708 loops=1)
                                                                                        Hash Cond: (aa2.ty = sign.aa)
                                                                                        ->  Hash Left Join  (cost=424275.90..462337.40 rows=28593 width=319) (actual time=3813.019..15036.693 rows=1175708 loops=1)
                                                                                              Hash Cond: (aa2.travdets = travdets.idnumber)
                                                                                              ->  Hash Left Join  (cost=408250.71..445773.33 rows=28593 width=307) (actual time=3559.650..12275.923 rows=1175708 loops=1)"
                                                                                                    Hash Cond: (aa2.aid = tblapplic.idnumber)
                                                                                                    ->  Hash Right Join  (cost=301618.16..338733.36 rows=28593 width=287) (actual time=2854.476..9334.815 rows=1175708 loops=1)
                                                                                                          Hash Cond: ((pm.aa2id = aa2.id) AND (pm.pagee = aa2.ty))
                                                                                                          ->  Seq Scan on pm  (cost=0.00..23718.76 rows=595176 width=22) (actual time=0.148..270.163 rows=583001 loops=1)
                                                                                                          ->  Hash  (cost=301189.26..301189.26 rows=28593 width=275) (actual time=2854.215..2854.215 rows=1167712 loops=1)
                                                                                                                Buckets: 4096  Batches: 4 (originally 1)  Memory Usage: 65537kB
                                                                                                                ->  Seq Scan on aa2  (cost=0.00..301189.26 rows=28593 width=275) (actual time=0.010..1806.989 rows=1167712 loops=1)
                                                                                                                      Filter: ((ty > 0) AND (op_status = 0))"
                                                                                                                      Rows Removed by Filter: 98527
                                                                                                    ->  Hash  (cost=98044.47..98044.47 rows=687047 width=28) (actual time=705.012..705.012 rows=447141 loops=1)
                                                                                                          Buckets: 131072  Batches: 1  Memory Usage: 28576kB"
                                                                                                          ->  Seq Scan on tblapplic  (cost=0.00..98044.47 rows=687047 width=28) (actual time=0.012..366.413 rows=447141 loops=1)
                                                                                              ->  Hash  (cost=10560.64..10560.64 rows=437164 width=20) (actual time=253.078..253.078 rows=419170 loops=1)
                                                                                                    Buckets: 65536  Batches: 1  Memory Usage: 22917kB"
                                                                                                    ->  Seq Scan on travdets  (cost=0.00..10560.64 rows=437164 width=20) (actual time=0.006..122.472 rows=419170 loops=1)
                                                                                        ->  Hash  (cost=1.20..1.20 rows=20 width=19) (actual time=0.014..0.014 rows=20 loops=1)"
                                                                                              Buckets: 1024  Batches: 1  Memory Usage: 2kB"
                                                                                              ->  Seq Scan on sign  (cost=0.00..1.20 rows=20 width=19) (actual time=0.004..0.009 rows=20 loops=1)
                                                                                  ->  Hash  (cost=5.53..5.53 rows=253 width=20) (actual time=0.143..0.143 rows=253 loops=1)
                                                                                        Buckets: 1024  Batches: 1  Memory Usage: 14kB
                                                                                        ->  Seq Scan on tblnationality dcou  (cost=0.00..5.53 rows=253 width=20) (actual time=0.007..0.064 rows=253 loops=1)
                                                                            ->  Hash  (cost=19.90..19.90 rows=290 width=31) (actual time=0.189..0.189 rows=294 loops=1)
                                                                                  Buckets: 1024  Batches: 1  Memory Usage: 20kB"
                                                                                  ->  Seq Scan on tblcountry  (cost=0.00..19.90 rows=290 width=31) (actual time=0.004..0.094 rows=294 loops=1)
                                                                      ->  Hash  (cost=17.76..17.76 rows=176 width=16) (actual time=0.576..0.576 rows=431 loops=1)
                                                                            Buckets: 1024  Batches: 1  Memory Usage: 21kB
                                                                            ->  Seq Scan on aa2pre  (cost=0.00..17.76 rows=176 width=16) (actual time=0.030..0.438 rows=494 loops=1)
                                                                ->  Hash  (cost=28.48..28.48 rows=548 width=16) (actual time=0.623..0.623 rows=507 loops=1)
                                                                      Buckets: 1024  Batches: 1  Memory Usage: 22kB
                                                                      ->  Seq Scan on tbluser  (cost=0.00..28.48 rows=548 width=16) (actual time=0.005..0.526 rows=507 loops=1)
                                                          ->  Hash  (cost=1.14..1.14 rows=14 width=17) (actual time=0.010..0.010 rows=13 loops=1)"
                                                                Buckets: 1024  Batches: 1  Memory Usage: 1kB"
                                                                ->  Seq Scan on loffice  (cost=0.00..1.14 rows=14 width=17) (actual time=0.003..0.005 rows=13 loops=1)"
                                                    ->  Hash  (cost=1.14..1.14 rows=14 width=17) (actual time=0.008..0.008 rows=13 loops=1)
                                                          Buckets: 1024  Batches: 1  Memory Usage: 1kB
                                                          ->  Seq Scan on loffice lofficesuperuser  (cost=0.00..1.14 rows=14 width=17) (actual time=0.002..0.005 rows=13 loops=1)
                                              ->  Hash  (cost=55.09..55.09 rows=1709 width=49) (actual time=0.998..0.998 rows=1717 loops=1)
                                                    Buckets: 1024  Batches: 1  Memory Usage: 144kB
                                                    ->  Seq Scan on stype  (cost=0.00..55.09 rows=1709 width=49) (actual time=0.005..0.431 rows=1717 loops=1)
                                        ->  Hash  (cost=5.40..5.40 rows=140 width=37) (actual time=0.100..0.100 rows=145 loops=1)
                                              Buckets: 1024  Batches: 1  Memory Usage: 11kB
                                              ->  Seq Scan on sta  (cost=0.00..5.40 rows=140 width=37) (actual time=0.004..0.046 rows=145 loops=1)
                                  ->  Materialize  (cost=0.00..1.10 rows=7 width=44) (actual time=0.000..0.001 rows=7 loops=1175709)
                                        ->  Seq Scan on tblcolltype  (cost=0.00..1.07 rows=7 width=44) (actual time=0.013..0.014 rows=7 loops=1)
                            ->  Hash  (cost=1.18..1.18 rows=18 width=27) (actual time=0.012..0.012 rows=18 loops=1)
                                  Buckets: 1024  Batches: 1  Memory Usage: 2kB"
                                  ->  Seq Scan on tbllist1  (cost=0.00..1.18 rows=18 width=27) (actual time=0.004..0.007 rows=18 loops=1)
                      ->  Hash  (cost=29.18..29.18 rows=818 width=34) (actual time=0.469..0.469 rows=827 loops=1)"
                            Buckets: 1024  Batches: 1  Memory Usage: 57kB"
                            ->  Seq Scan on tblreferral  (cost=0.00..29.18 rows=818 width=34) (actual time=0.007..0.181 rows=827 loops=1)
                ->  Hash  (cost=3.88..3.88 rows=188 width=23) (actual time=0.099..0.099 rows=191 loops=1)
                      Buckets: 1024  Batches: 1  Memory Usage: 11kB
                      ->  Seq Scan on namess  (cost=0.00..3.88 rows=188 width=23) (actual time=0.007..0.042 rows=191 loops=1)
          ->  Hash  (cost=23.10..23.10 rows=1310 width=16) (actual time=0.000..0.000 rows=0 loops=1)
                Buckets: 1024  Batches: 1  Memory Usage: 0kB
                ->  Seq Scan on aa2_curr_act curact  (cost=0.00..23.10 rows=1310 width=16) (actual time=0.000..0.000 rows=0 loops=1)
          SubPlan 2
            ->  Result  (cost=20.66..20.67 rows=1 width=0) (actual time=0.156..0.156 rows=1 loops=1175709)
                  InitPlan 1 (returns $1)
                    ->  Sort  (cost=20.65..20.66 rows=3 width=23) (actual time=0.135..0.137 rows=25 loops=1175709)
                          Sort Key: public.tblcountry.descr
                          Sort Method: quicksort  Memory: 28kB
                          ->  Seq Scan on tblcountry  (cost=0.00..20.62 rows=3 width=23) (actual time=0.017..0.063 rows=25 loops=1175709)
                                Filter: (dcountry = travdets.dcountry)
                                Rows Removed by Filter: 269
          SubPlan 4
            ->  Result  (cost=44.29..44.31 rows=1 width=0) (actual time=0.237..0.237 rows=1 loops=1175709)
                  InitPlan 3 (returns $4)
                    ->  Sort  (cost=44.29..44.29 rows=1 width=41) (actual time=0.224..0.225 rows=12 loops=1175709)
                          Sort Key: public.stype.descr
                          Sort Method: quicksort  Memory: 25kB
                          ->  Bitmap Heap Scan on stype  (cost=4.93..44.28 rows=1 width=41) (actual time=0.108..0.197 rows=12 loops=1175709)
                                Recheck Cond: (typee = aa2.ty)
                                Filter: (visaa = COALESCE(aa2.visaid, 0::bigint))
                                Rows Removed by Filter: 629
                                ->  Bitmap Index Scan on stypeindtypee  (cost=0.00..4.93 rows=90 width=0) (actual time=0.073..0.073 rows=745 loops=1175709)
                                      Index Cond: (typee = aa2.ty)
          SubPlan 6
            ->  Result  (cost=5.87..5.88 rows=1 width=0) (actual time=0.056..0.056 rows=1 loops=1175709)
                  InitPlan 5 (returns $6)
                    ->  Sort  (cost=5.85..5.87 rows=7 width=27) (actual time=0.044..0.045 rows=13 loops=1175709)
                          Sort Key: public.sta.descr
                          Sort Method: quicksort  Memory: 25kB
                          ->  Seq Scan on sta  (cost=0.00..5.75 rows=7 width=27) (actual time=0.010..0.025 rows=13 loops=1175709)
                                Filter: (typee = aa2.ty)
                                Rows Removed by Filter: 132
Total runtime: 569133.611 ms

最新编辑:

对于那些没有时间解决问题的人来说,这是一种解决方法。只需包装大查询,然后进行ORDER BY。像这样:

explain ANALYSE
SELECT * FROM
(select aa2.aid as cid
,aa2.id ,aa2.ty 
,coalesce(sign.bb,'') as tyy
,aa2.rd ,aa2.dt ,aa2.appliedon 
,aa2.approvedon ,aa2.pprec 
,aa2.ppret ,aa2.cdate ,pm.pmd,tblapplic.dob 
,coalesce(tblapplic.firstname ,'') as fname 
,coalesce(tblapplic.middlename ,'') as mname 
,coalesce(tblapplic.surname ,'') as sname 
,coalesce(Upper(dcou.descr),'') as dcouu 
,coalesce(aa2.visaid,0) as visaid 
,coalesce(tblcountry.descr,'') as visaname 
,(SELECT chr(9) || array_to_string(ARRAY(SELECT descr FROM tblcountry WHERE dcountry=travdets.dcountry ORDER BY descr),chr(9))) as visanamecombo 
,coalesce(aa2.stype,0) as stypeid ,coalesce(stype.descr,'') as stype 
,coalesce(aa2.sts,0) as appStatusid ,coalesce(sta.descr,'') as appStatus 
,lofficesuperuser.descr as lofficesuperuser,coalesce(aa2.loffice,0) as lofficeid 
,coalesce(lofficesuperuser.descr,coalesce(loffice.descr,'')) as loffice 
,coalesce(pm.id,0) as pmid 
,(SELECT chr(9) || array_to_string(ARRAY(SELECT descr FROM stype WHERE typee=aa2.ty and visaa=coalesce(aa2.visaid,0) ORDER BY descr),chr(9))) as sTypeCombo 
,(SELECT chr(9) || array_to_string(ARRAY(SELECT descr FROM sta WHERE typee=aa2.ty ORDER BY descr),chr(9))) as appStatusCombo 
,coalesce(aa2.colltype,0) as colltypeid
,coalesce(tblcolltype.descr,'') as colltype,aa2.embcolldt as embcolldt
,aa2.clcolldt as clbcolldt,travdets.leavingdt as leaving,coalesce(aa2.pptrnum,'') as ppTrackNumber
,coalesce(aa2.rt,0) as regtypeid,coalesce(tbllist1.descr,'') as regtype
,coalesce(aa2.reff,0) as referralid ,coalesce(tblreferral.descr,'') as referral
,coalesce(aa2.scaller,0) as callerid 
,coalesce(namess.descr,'') as caller,coalesce(aa2.travdets,0) as travdetsid
,coalesce(curact.luser,-1) as actionluser
,aa2.iss 
from aa2  
LEFT JOIN tblapplic on aa2.aid=tblapplic.idnumber  
left JOIN travdets on aa2.travdets=travdets.idnumber  
left JOIN pm on aa2.id=pm.aa2id  and pm.pagee=aa2.ty 
LEFT JOIN sign ON aa2.ty = sign.aa  
LEFT JOIN tblnationality dcou ON dcou.idnumber=travdets.dcountry 
left join tblcountry on aa2.visaid=tblcountry.idnumber  
left join aa2pre on aa2.id=aa2pre.aa2id 
left join tbluser on aa2pre.addusr=tbluser.idnumber  
left join loffice on tbluser.office=loffice.idnumber  
left join loffice lofficesuperuser on aa2.loffice=lofficesuperuser.idnumber  
left outer join stype on aa2.stype=stype.id  
left outer join sta on aa2.sts=sta.idnumber and sta.typee=aa2.ty 
left join tblcolltype on aa2.colltype=tblcolltype.idnumber  
left join tbllist1 on aa2.rt=tbllist1.idnumber  
left join tblreferral on aa2.reff=tblreferral.idnumber  
left join namess on aa2.scaller=namess.idnumber  
left join aa2_curr_act curact on aa2.id=curact.aa2id 
where aa2.op_status=0 and aa2.ty>0)
ORDER BY aa2.aid DESC, aa2.ty DESC 
LIMIT 1000

3 个答案:

答案 0 :(得分:1)

这背后的原因是您的查询没有使用任何表的索引。

  

对于需要扫描表的大部分内容的查询,显式排序可能比使用索引更快,因为它遵循顺序访问模式需要更少的磁盘I / O.当只需要提取几行时,索引更有用。 1

如果ORDERY BY可以使用索引,您会在EXPLAIN中看到类似的内容:

Index Scan using test_idx on test_tbl (cost=0.15..24.28 rows=285 width=27)

F.ex。在我的(较小的)表中使用此索引,如果使用了LIMIT 210,但使用了整个表扫描,如果我提供LIMIT 211(您的表可能不同,查询计划程序可以使用行号& width考虑到f.ex.,同时决定使用索引,或不)。

答案 1 :(得分:1)

  • [我假设您的数据模型具有主键,外键和外键的适当索引]

  • 对于查询的内部部分,预期行数和观察行数之间似乎存在差异,表明统计信息不存在或已过期。这可能导致计划程序选择散列连接,但其大小可能导致散列表溢出到磁盘。尝试在受影响的表上运行VACUUM ANALYZE tbl_name;,这将刷新统计信息。

  • 如果你有足够的内存而没有太多的并发会话你可以尝试将work_mem设置得更高,这个可以使引擎能够将其哈希表保持在核心

  • 连接表的数量相当大,可能大于join_collapse_limit(默认设置为8)。尝试将其设置得更高,这可以使优化器超越其当前的范围。

  • 开始调整后,尝试将effective_cach_size设置为至少物理内存的一半,将random_page_cost设置为低于默认值4的值,(尝试1.5)

在可能的情况下,此调优将有利于散列连接之上的索引连接。

[可以通过发出SET parameter_name = value;在当前.sql中设置所有配置设置]更新统计信息是永久性的,但不会造成任何伤害。

答案 2 :(得分:0)

对于没有时间解决问题的用户,这里是一种解决方法。只需包装大查询,然后进行ORDER BY。