使用多态实体进行查询优化

时间:2018-03-03 02:11:34

标签: postgresql

我正在尝试优化一个复杂的查询,其中包括2个多态实体( materialable markertable )。 Materialable 中有5种案例,可销售有4种案例。

因此,在这种情况下的目标是使用这两个多态实体作为连接器将回扣(~600k记录)表与 xin_demand_plan_shiptos (~3M记录)表连接起来。这是通过下面的查询实现的。完成大约需要2分钟。

经过无数次尝试和错误优化后,我尝试将查询拆分为20个相同的查询(= 5 x 4),这些查询涵盖了这些实体上所有可能的多态组合,并将其结果与19 UNION ALL相关联。执行时间从2分钟下降到1.2秒(优化器耗尽1.0秒)。

这让我觉得必须有一种方法可以重新重写原始查询和/或添加或删除索引或优化选项,以获得相同的1.2秒而无需复制查询20次。我欢迎并感谢任何想法

ORIGINAL QUERY:

select
    r.id            
    , c.period 
    , xdps.sku_id 
    , xdps.sold_to_point_id_indirect    
    , xdps.ship_to_point_id             
    , sum(xdps.forecast_units)          
    , sum(xdps.forecast_dollars)        
    , r.oi                              
    , r.rebate_type                     
    , r."value"                         
    , r.net_price                       
FROM rebates r 
INNER JOIN events e                     
    ON e.id = r.event_id
INNER JOIN markets m                    
    ON m.event_id = e.id
INNER JOIN calendar c                   
    ON c.type = 'monthly' 
        AND (c."to", c."from") overlaps (r.ship_start,r.ship_end)
INNER JOIN xin_demand_plan_shiptos xdps ON
        (
                (r.materialable_id = sku_id                 and materialable_type = 'Sku') 
            or  (r.materialable_id = promo_group_id         and materialable_type = 'PromoGroup') 
            or  (r.materialable_id = brand_id               and materialable_type = 'Brand') 
            or  (r.materialable_id = product_level_2_id     and materialable_type = 'ProductLevel2') 
            or  (r.materialable_id = lob_id                 and materialable_type = 'Lob')
        )           
    AND (r.lob_constraint = xdps.lob_id OR r.lob_constraint IS NULL)
    AND  
        (
                (m.marketable_id = xdps.sold_to_point_id_indirect AND m.marketable_type = 'SoldtoPoint')
            or  (m.marketable_id = xdps.sold_to_group_id_indirect AND m.marketable_type = 'SoldtoGroup')
            or  (m.marketable_id = xdps.ship_to_point_id          AND m.marketable_type = 'ShiptoPoint')
            or  (m.marketable_id = xdps.shipto_group_id           AND m.marketable_type = 'ShiptoGroup')
        )
    AND xdps."period"                               =  c."from"
    AND xdps."period" + INTERVAL '1 MONTH - 1 DAY'  >= r.ship_start
    AND xdps."period"                               <= r.ship_end
WHERE       r.pos                       IS FALSE 
            AND r.id = any(array((select array_agg(id) from rebates where event_id = 24447)))  
GROUP BY    r.id
            , c.period
            , xdps.sku_id
            , e.beneficiary_definer 
            , e.beneficiary_if_someone_else
            , xdps.ship_to_point_id
            , xdps.shipto_group_id
            , xdps.sold_to_group_id_indirect
            , xdps.sold_to_point_id_indirect

原始查询的EXPLAIN ANALYZE:

HashAggregate  (cost=133628.70..133628.79 rows=4 width=81) (actual time=117542.764..117622.281 rows=70895 loops=1)
  Group Key: r.id, c.period, xdps.sku_id, e.beneficiary_definer, e.beneficiary_if_someone_else, xdps.ship_to_point_id, xdps.shipto_group_id, xdps.sold_to_group_id_indirect, xdps.sold_to_point_id_indirect
  InitPlan 1 (returns $0)
    ->  Aggregate  (cost=59.74..59.75 rows=1 width=4) (actual time=1.132..1.133 rows=1 loops=1)
          ->  Index Scan using index_rebates_on_event_id on rebates  (cost=0.55..59.62 rows=44 width=4) (actual time=0.015..0.780 rows=751 loops=1)
                Index Cond: (event_id = 24447)
  ->  Hash Join  (cost=3642.86..133568.85 rows=4 width=81) (actual time=2979.937..117441.543 rows=70895 loops=1)
        Hash Cond: (xdps.period = c."from")
        Join Filter: "overlaps"((c."to")::timestamp with time zone, (c."from")::timestamp with time zone, (r.ship_start)::timestamp with time zone, (r.ship_end)::timestamp with time zone)
        ->  Nested Loop  (cost=3636.06..133561.70 rows=13 width=85) (actual time=2979.712..117341.178 rows=70895 loops=1)
              Join Filter: ((((r.lob_constraint)::text = (xdps.lob_id)::text) OR (r.lob_constraint IS NULL)) AND (xdps.period <= r.ship_end) AND ((xdps.period + '1 mon -1 days'::interval) >= r.ship_start) AND ((((r.materialable_id)::text = (xdps.sku_id)::text) AND ((r.materialable_type)::text = 'Sku'::text)) OR (((r.materialable_id)::text = (xdps.promo_group_id)::text) AND ((r.materialable_type)::text = 'PromoGroup'::text)) OR (((r.materialable_id)::text = (xdps.brand_id)::text) AND ((r.materialable_type)::text = 'Brand'::text)) OR (((r.materialable_id)::text = (xdps.product_level_2_id)::text) AND ((r.materialable_type)::text = 'ProductLevel2'::text)) OR (((r.materialable_id)::text = (xdps.lob_id)::text) AND ((r.materialable_type)::text = 'Lob'::text))))
              Rows Removed by Join Filter: 61793481
              ->  Nested Loop  (cost=1.25..54.25 rows=5 width=140) (actual time=1.275..29.227 rows=751 loops=1)
                    ->  Nested Loop  (cost=0.84..45.11 rows=7 width=131) (actual time=1.266..20.148 rows=751 loops=1)
                          ->  Index Scan using rebates_pkey on rebates r  (cost=0.42..26.61 rows=7 width=119) (actual time=1.259..11.174 rows=751 loops=1)
                                Index Cond: (id = ANY ($0))
                                Filter: ((pos IS FALSE) AND (rebate_type = ANY ('{0,1,2,3}'::integer[])) AND (((materialable_type)::text = 'Sku'::text) OR ((materialable_type)::text = 'PromoGroup'::text) OR ((materialable_type)::text = 'Brand'::text) OR ((materialable_type)::text = 'ProductLevel2'::text) OR ((materialable_type)::text = 'Lob'::text)))
                          ->  Index Scan using index_events_on_id_readable_id on events e  (cost=0.41..2.63 rows=1 width=12) (actual time=0.007..0.008 rows=1 loops=751)
                                Index Cond: (id = r.event_id)
                    ->  Index Only Scan using markets_event_id_marketable_id_marketable_type_idx on markets m  (cost=0.41..1.30 rows=1 width=21) (actual time=0.007..0.009 rows=1 loops=751)
                          Index Cond: (event_id = e.id)
                          Filter: (((marketable_type)::text = 'SoldtoPoint'::text) OR ((marketable_type)::text = 'SoldtoGroup'::text) OR ((marketable_type)::text = 'ShiptoPoint'::text) OR ((marketable_type)::text = 'ShiptoGroup'::text))
                          Heap Fetches: 0
              ->  Bitmap Heap Scan on xin_demand_plan_shiptos xdps  (cost=3634.81..26695.87 rows=125 width=85) (actual time=11.370..85.191 rows=82376 loops=751)
                    Recheck Cond: (((m.marketable_id)::text = (sold_to_point_id_indirect)::text) OR ((m.marketable_id)::text = (sold_to_group_id_indirect)::text) OR ((m.marketable_id)::text = (ship_to_point_id)::text) OR ((m.marketable_id)::text = (shipto_group_id)::text))
                    Filter: ((((m.marketable_type)::text = 'SoldtoPoint'::text) AND ((m.marketable_id)::text = (sold_to_point_id_indirect)::text)) OR (((m.marketable_type)::text = 'SoldtoGroup'::text) AND ((m.marketable_id)::text = (sold_to_group_id_indirect)::text)) OR (((m.marketable_type)::text = 'ShiptoPoint'::text) AND ((m.marketable_id)::text = (ship_to_point_id)::text)) OR (((m.marketable_type)::text = 'ShiptoGroup'::text) AND ((m.marketable_id)::text = (shipto_group_id)::text)))
                    Heap Blocks: exact=6970031
                    ->  BitmapOr  (cost=3634.81..3634.81 rows=25011 width=0) (actual time=9.862..9.862 rows=0 loops=751)
                          ->  Bitmap Index Scan on xin_demand_plan_shiptos_sold_to_point_id_indirect_idx  (cost=0.00..19.46 rows=1766 width=0) (actual time=0.007..0.007 rows=0 loops=751)
                                Index Cond: ((m.marketable_id)::text = (sold_to_point_id_indirect)::text)
                          ->  Bitmap Index Scan on xin_demand_plan_shiptos_sold_to_group_id_indirect_idx  (cost=0.00..79.11 rows=8178 width=0) (actual time=6.457..6.457 rows=82376 loops=751)
                                Index Cond: ((m.marketable_id)::text = (sold_to_group_id_indirect)::text)
                          ->  Bitmap Index Scan on index_xin_demand_plan_shiptos_on_ship_to_point_id  (cost=0.00..444.66 rows=1439 width=0) (actual time=0.008..0.008 rows=0 loops=751)
                                Index Cond: ((m.marketable_id)::text = (ship_to_point_id)::text)
                          ->  Bitmap Index Scan on index_xin_demand_plan_shiptos_on_shipto_group_id  (cost=0.00..3091.45 rows=13627 width=0) (actual time=3.384..3.384 rows=45284 loops=751)
                                Index Cond: ((m.marketable_id)::text = (shipto_group_id)::text)
        ->  Hash  (cost=4.40..4.40 rows=192 width=16) (actual time=0.205..0.205 rows=192 loops=1)
              Buckets: 1024  Batches: 1  Memory Usage: 17kB
              ->  Seq Scan on calendar c  (cost=0.00..4.40 rows=192 width=16) (actual time=0.008..0.126 rows=192 loops=1)
                    Filter: ((type)::text = 'monthly'::text)
Planning time: 8.031 ms
Execution time: 117649.466 ms

重写为20 UNION ALL查询时查询的EXPLAIN ANALYZE: 只显示20个中的一个子计划(它们看起来相同)但是这个为这个特定的多态组合做了所有的工作

    Append  (cost=3207.43..123158.92 rows=22 width=152) (actual time=551.503..764.736 rows=70895 loops=1)                                                                                                                                                         
  ->  Subquery Scan on "*SELECT* 6"  (cost=76.85..76.88 rows=1 width=152) (actual time=18.680..18.749 rows=45 loops=1)                                                                                                                                         
        ->  HashAggregate  (cost=76.85..76.87 rows=1 width=81) (actual time=18.678..18.713 rows=45 loops=1)                                                                                                                                                    
              Group Key: r_5.id, c_5.period, xdps_5.sku_id, e_5.beneficiary_definer, e_5.beneficiary_if_someone_else, xdps_5.ship_to_point_id, xdps_5.shipto_group_id, xdps_5.sold_to_group_id_indirect, xdps_5.sold_to_point_id_indirect                      
              InitPlan 6 (returns $25)                                                                                                                                                                                                                         
                ->  Aggregate  (cost=18.75..18.76 rows=1 width=4) (actual time=0.672..0.672 rows=1 loops=1)                                                                                                                                                    
                      ->  Index Only Scan using rebates_event_rebate_idx on rebates rebates_5  (cost=0.42..16.87 rows=751 width=4) (actual time=0.015..0.343 rows=751 loops=1)                                                                                 
                            Index Cond: (event_id = 24447)                                                                                                                                                                                                     
                            Heap Fetches: 0                                                                                                                                                                                                                    
              ->  Nested Loop  (cost=1.83..58.07 rows=1 width=81) (actual time=17.663..18.595 rows=45 loops=1)                                                                                                                                                 
                    Join Filter: "overlaps"((c_5."to")::timestamp with time zone, (c_5."from")::timestamp with time zone, (r_5.ship_start)::timestamp with time zone, (r_5.ship_end)::timestamp with time zone)                                                
                    ->  Nested Loop  (cost=1.69..57.88 rows=1 width=85) (actual time=17.648..18.407 rows=45 loops=1)                                                                                                                                           
                          Join Filter: (r_5.event_id = e_5.id)                                                                                                                                                                                                 
                          ->  Nested Loop  (cost=1.27..56.37 rows=1 width=85) (actual time=17.635..18.212 rows=45 loops=1)                                                                                                                                     
                                ->  Nested Loop  (cost=0.84..37.38 rows=5 width=57) (actual time=0.706..5.561 rows=751 loops=1)                                                                                                                                
                                      ->  Index Scan using rebates_pkey on rebates r_5  (cost=0.42..26.50 rows=7 width=47) (actual time=0.699..2.776 rows=751 loops=1)                                                                                         
                                            Index Cond: (id = ANY ($25))                                                                                                                                                                                       
                                            Filter: ((pos IS FALSE) AND ((materialable_type)::text = 'Sku'::text) AND (rebate_type = ANY ('{0,1,2,3}'::integer[])))                                                                                            
                                      ->  Index Only Scan using markets_event_id_marketable_id_marketable_type_idx on markets m_5  (cost=0.41..1.55 rows=1 width=10) (actual time=0.002..0.002 rows=1 loops=751)                                               
                                            Index Cond: ((event_id = r_5.event_id) AND (marketable_type = 'SoldtoGroup'::text))                                                                                                                                
                                            Heap Fetches: 0                                                                                                                                                                                                    
                                ->  Index Scan using xin_demand_plan_shiptos_sold_to_group_id_indirect_sku_id_idx on xin_demand_plan_shiptos xdps_5  (cost=0.43..3.79 rows=1 width=59) (actual time=0.015..0.016 rows=0 loops=751)                             
                                      Index Cond: (((sold_to_group_id_indirect)::text = (m_5.marketable_id)::text) AND ((sku_id)::text = (r_5.materialable_id)::text))                                                                                         
                                      Filter: ((((r_5.lob_constraint)::text = (lob_id)::text) OR (r_5.lob_constraint IS NULL)) AND (period <= r_5.ship_end) AND ((period + '1 mon -1 days'::interval) >= r_5.ship_start))                                      
                                      Rows Removed by Filter: 11                                                                                                                                                                                               
                          ->  Index Scan using events_pkey on events e_5  (cost=0.41..1.49 rows=1 width=12) (actual time=0.002..0.002 rows=1 loops=45)                                                                                                         
                                Index Cond: (id = m_5.event_id)                                                                                                                                                                                                
                    ->  Index Scan using "from+period" on calendar c_5  (cost=0.14..0.17 rows=1 width=16) (actual time=0.001..0.002 rows=1 loops=45)                                                                                                           
                          Index Cond: ("from" = xdps_5.period)                                                                                                                                                                                                                      

1 个答案:

答案 0 :(得分:0)

经过更多的实验,我以某种方式将原始查询的时间缩短到80ms。我不确定究竟是什么工作,但我怀疑这是我使用逻辑路径构建的几个多列索引的引入,如果我&#34;是#34;优化器,使用我的直觉来获取底层数据。有趣的是,如果我取出行AND r.rebate_type IN (0,1,2,3),这对于此测试用例的结果应该是无关紧要的,执行时间会跳到5秒。

结果查询:

explain analyze
select
    r.id            
    , c.period 
    , xdps.sku_id 
    , xdps.sold_to_point_id_indirect    
    , xdps.ship_to_point_id             
    , sum(xdps.forecast_units)          
    , sum(xdps.forecast_dollars)        
    , r.oi                              
    , r.rebate_type                     
    , r."value"                         
    , r.net_price                       
FROM rebates r 
INNER JOIN events e     ON e.id = r.event_id
INNER JOIN markets m    ON m.event_id = e.id
INNER JOIN calendar c   ON c.type = 'monthly' AND c."to" >= r.ship_start and c.from <= r.ship_end
INNER JOIN xin_demand_plan_shiptos xdps ON
        (
                (r.materialable_id = sku_id                 and materialable_type = 'Sku') 
            or  (r.materialable_id = promo_group_id         and materialable_type = 'PromoGroup') 
            or  (r.materialable_id = brand_id               and materialable_type = 'Brand') 
            or  (r.materialable_id = product_level_2_id     and materialable_type = 'ProductLevel2') 
            or  (r.materialable_id = lob_id                 and materialable_type = 'Lob')
        )           
    AND (r.lob_constraint = xdps.lob_id OR r.lob_constraint IS NULL)
    AND  
        (
                (m.marketable_id = xdps.sold_to_point_id_indirect AND m.marketable_type = 'SoldtoPoint')
            or  (m.marketable_id = xdps.sold_to_group_id_indirect AND m.marketable_type = 'SoldtoGroup')
            or  (m.marketable_id = xdps.ship_to_point_id          AND m.marketable_type = 'ShiptoPoint')
            or  (m.marketable_id = xdps.shipto_group_id           AND m.marketable_type = 'ShiptoGroup')
        )
    AND xdps."period"                               =  c."from"
    AND xdps."period" + INTERVAL '1 MONTH - 1 DAY'  >= r.ship_start
    AND xdps."period"                               <= r.ship_end
WHERE       r.pos               IS false
            AND r.rebate_type IN (0,1,2,3) --including lump_sum
            AND r.event_id =    24447  
GROUP BY    r.id
            , c.period
            , xdps.sku_id
            , e.beneficiary_definer 
            , e.beneficiary_if_someone_else
            , xdps.ship_to_point_id
            , xdps.shipto_group_id
            , xdps.sold_to_group_id_indirect
            , xdps.sold_to_point_id_indirect

其解析分析:

HashAggregate  (cost=15038.39..15038.84 rows=30 width=80) (actual time=82.779..82.814 rows=45 loops=1)
  Group Key: r.id, c.period, xdps.sku_id, e.beneficiary_definer, e.beneficiary_if_someone_else, xdps.ship_to_point_id, xdps.shipto_group_id, xdps.sold_to_group_id_indirect, xdps.sold_to_point_id_indirect
  ->  Merge Join  (cost=11688.17..15037.57 rows=30 width=80) (actual time=80.854..82.702 rows=45 loops=1)
        Merge Cond: (c."from" = xdps.period)
        Join Filter: ((((r.lob_constraint)::text = (xdps.lob_id)::text) OR (r.lob_constraint IS NULL)) AND (xdps.period <= r.ship_end) AND ((xdps.period + '1 mon -1 days'::interval) >= r.ship_start) AND ((((r.materialable_id)::text = (xdps.sku_id)::text) AND ((r.materialable_type)::text = 'Sku'::text)) OR (((r.materialable_id)::text = (xdps.promo_group_id)::text) AND ((r.materialable_type)::text = 'PromoGroup'::text)) OR (((r.materialable_id)::text = (xdps.brand_id)::text) AND ((r.materialable_type)::text = 'Brand'::text)) OR (((r.materialable_id)::text = (xdps.product_level_2_id)::text) AND ((r.materialable_type)::text = 'ProductLevel2'::text)) OR (((r.materialable_id)::text = (xdps.lob_id)::text) AND ((r.materialable_type)::text = 'Lob'::text))))
        Rows Removed by Join Filter: 1117
        ->  Nested Loop  (cost=11.52..2672.96 rows=11883 width=63) (actual time=30.190..30.367 rows=2 loops=1)
              Join Filter: ((c."to" >= r.ship_start) AND (c."from" <= r.ship_end))
              Rows Removed by Join Filter: 36047
              ->  Index Scan using "from" on calendar c  (cost=0.14..7.80 rows=192 width=16) (actual time=0.012..0.056 rows=49 loops=1)
                    Filter: ((type)::text = 'monthly'::text)
              ->  Materialize  (cost=11.37..795.03 rows=557 width=51) (actual time=0.003..0.298 rows=736 loops=49)
                    ->  Bitmap Heap Scan on rebates r  (cost=11.37..792.25 rows=557 width=51) (actual time=0.120..1.140 rows=751 loops=1)
                          Recheck Cond: (event_id = 24447)
                          Filter: ((pos IS FALSE) AND (rebate_type = ANY ('{0,1,2,3}'::integer[])) AND (((materialable_type)::text = 'Sku'::text) OR ((materialable_type)::text = 'PromoGroup'::text) OR ((materialable_type)::text = 'Brand'::text) OR ((materialable_type)::text = 'ProductLevel2'::text) OR ((materialable_type)::text = 'Lob'::text)))
                          Heap Blocks: exact=312
                          ->  Bitmap Index Scan on rebates_event_id_rebate_type_pos_oi_idx  (cost=0.00..11.24 rows=716 width=0) (actual time=0.081..0.081 rows=751 loops=1)
                                Index Cond: ((event_id = 24447) AND (pos = false))
        ->  Sort  (cost=11165.28..11167.73 rows=978 width=96) (actual time=43.627..47.199 rows=8873 loops=1)
              Sort Key: xdps.period
              Sort Method: quicksort  Memory: 1632kB
              ->  Nested Loop  (cost=125.70..11116.71 rows=978 width=96) (actual time=3.979..36.881 rows=8873 loops=1)
                    ->  Nested Loop  (cost=0.83..4.21 rows=1 width=29) (actual time=0.029..0.034 rows=1 loops=1)
                          ->  Index Scan using index_events_on_id_readable_id on events e  (cost=0.41..2.66 rows=1 width=12) (actual time=0.014..0.016 rows=1 loops=1)
                                Index Cond: (id = 24447)
                          ->  Index Only Scan using markets_event_id_marketable_id_marketable_type_idx on markets m  (cost=0.41..1.54 rows=1 width=21) (actual time=0.011..0.014 rows=1 loops=1)
                                Index Cond: (event_id = 24447)
                                Filter: (((marketable_type)::text = 'SoldtoPoint'::text) OR ((marketable_type)::text = 'SoldtoGroup'::text) OR ((marketable_type)::text = 'ShiptoPoint'::text) OR ((marketable_type)::text = 'ShiptoGroup'::text))
                                Heap Fetches: 0
                    ->  Bitmap Heap Scan on xin_demand_plan_shiptos xdps  (cost=124.87..11111.93 rows=57 width=84) (actual time=3.944..29.858 rows=8873 loops=1)
                          Recheck Cond: (((m.marketable_id)::text = (sold_to_point_id_indirect)::text) OR ((m.marketable_id)::text = (sold_to_group_id_indirect)::text) OR ((m.marketable_id)::text = (ship_to_point_id)::text) OR ((m.marketable_id)::text = (shipto_group_id)::text))
                          Filter: ((((m.marketable_id)::text = (sold_to_point_id_indirect)::text) AND ((m.marketable_type)::text = 'SoldtoPoint'::text)) OR (((m.marketable_id)::text = (sold_to_group_id_indirect)::text) AND ((m.marketable_type)::text = 'SoldtoGroup'::text)) OR (((m.marketable_id)::text = (ship_to_point_id)::text) AND ((m.marketable_type)::text = 'ShiptoPoint'::text)) OR (((m.marketable_id)::text = (shipto_group_id)::text) AND ((m.marketable_type)::text = 'ShiptoGroup'::text)))
                          Rows Removed by Filter: 29424
                          Heap Blocks: exact=4278
                          ->  BitmapOr  (cost=124.87..124.87 rows=11426 width=0) (actual time=3.295..3.295 rows=0 loops=1)
                                ->  Bitmap Index Scan on xin_demand_plan_shiptos_sold_to_point_id_indirect_idx  (cost=0.00..6.97 rows=579 width=0) (actual time=0.011..0.011 rows=0 loops=1)
                                      Index Cond: ((m.marketable_id)::text = (sold_to_point_id_indirect)::text)
                                ->  Bitmap Index Scan on xin_demand_plan_shiptos_sold_to_group_id_indirect_idx  (cost=0.00..29.43 rows=2693 width=0) (actual time=0.878..0.878 rows=8873 loops=1)
                                      Index Cond: ((m.marketable_id)::text = (sold_to_group_id_indirect)::text)
                                ->  Bitmap Index Scan on index_xin_demand_plan_shiptos_on_ship_to_point_id  (cost=0.00..12.59 rows=1035 width=0) (actual time=0.008..0.008 rows=0 loops=1)
                                      Index Cond: ((m.marketable_id)::text = (ship_to_point_id)::text)
                                ->  Bitmap Index Scan on index_xin_demand_plan_shiptos_on_shipto_group_id  (cost=0.00..75.82 rows=7119 width=0) (actual time=2.391..2.391 rows=31428 loops=1)
                                      Index Cond: ((m.marketable_id)::text = (shipto_group_id)::text)
Planning time: 20.265 ms
Execution time: 82.994 ms