由于静态优化器策略导致的慢速参数化/过滤查询的Oracle SQL策略

时间:2014-03-13 12:44:22

标签: oracle10g sql-execution-plan union-all optimization optimizer-hints

比下面更简单地说:如果一个人有一个或多个查询参数,例如性能至关重要x_id ,(或报告/表格函数参数)<(例如,可以使用某些主键索引),可能 (取决于应用的用例/报告过滤器,...)

之一
  • null
  • 完全匹配(例如某些唯一ID)
  • like expression
  • 甚至是正则表达式

然后,如果所有这些可能性在单个查询中编码,我只会看到并知道优化程序

  • 生成唯一的静态计划,与实际参数运行时值
  • 无关
  • 因此不能假设在x_id上使用某些索引,尽管它可能是例如一些完全匹配

处理此问题的方式而不是

  1. 让一些PL / SQL代码选择n预定义和用例优化查询/视图?
    • 这个灵活的参数越多就越大
  2. 一些手动字符串构造和动态编译的查询?

  3. 基本上我有两个略有不同的用例/问题,如下所示:

    A - select * from tf_sel

    B - select * from data_union

    可能通过 SQL提示或使用其他技巧解决。

    为了加快这些查询速度,我目前正在某个实施级别(表格函数)上分离“合并查询”,这非常繁琐并且难以维护,但是由于更好的执行计划,确保查询运行得非常快。

    正如我所看到的,主要问题似乎是优化程序sql计划的静态性质总是相同的,尽管它可以更高效,如果它会考虑一些“查询时间 - 恒定的“滤波器参数。

    with
        -- Question A: What would be a good strategy to make tf_sel with tf_params nearly as fast as query_use_case_1_eq
        --             which actually provides the same result?
        --
        -- - a complex query should be used in various reports with filters
        -- - we want to keep as much as possible filter functionality on the db side (not the report engine side)
        --   to be able to utilize the fast and efficient db engine and for loosely coupled software design
    
    
        complex_query as (  -- just some imaginable complex query with a lot of table/view joins, aggregation/analytical functions etc.
            select 1 as id, 'ab12' as indexed_val, 'asdfasdf' x from dual
            union all select 2, 'ab34', 'a uiop345' from dual
            union all select 3, 'xy34', 'asdf  0u0duaf' from dual
            union all select 4, 'xy55', ' asdja´sf asd' from dual
        )
    
    
    -- <<< comment the following lines in to test it with the above
    
    --  , query_use_case_1_eq as (  -- quite fast and maybe the 95% use case
    --      select * from complex_query where indexed_val = 'ab12'
    --  )
    --select * from query_use_case_1_eq 
    
    -- >>>
    
    -- ID INDEXED_VAL X
    -- -- ----------- --------
    --  1 ab12        asdfasdf
    
    
    -- <<< comment the following lines in to test it with the above
    
    --  , query_use_case_2_all as (  -- significantly slower due to a lot of underlying calculations
    --      select * from complex_query
    --  )
    --select * from query_use_case_2_all
    
    -- >>>
    
    -- ID INDEXED_VAL X
    -- -- ----------- -------------
    --  1 ab12        asdfasdf
    --  2 ab34        a uiop345
    --  3 xy34        asdf  0u0duaf
    --  4 xy55         asdja´sf asd
    
    
    -- <<< comment the following lines in to test it with the above
    
    --  , query_use_case_3_like as (
    --      select * from complex_query where indexed_val like 'ab%'
    --  )
    --select * from query_use_case_3_like
    
    -- >>>
    
    -- ID INDEXED_VAL X
    -- -- ----------- ---------
    --  1 ab12        asdfasdf
    --  2 ab34        a uiop345
    
    
    -- <<< comment the following lines to simulate the table function
    
        , tf_params as (  -- table function params: imagine we have a table function where these are passed depending on the report
            select  'ab12' p_indexed_val,  'eq' p_filter_type  from dual
        )
        , tf_sel as (  -- table function select: nicely integrating all query possiblities, but beeing veeery slow :-(
            select q.* 
            from 
                tf_params p  -- just here so this example works without the need for the actual function
                join complex_query q on (1=1)
            where
                    p_filter_type = 'all'
                or (p_filter_type = 'eq' and indexed_val = p_indexed_val)
                or (p_filter_type = 'like' and indexed_val like p_indexed_val)
                or (p_filter_type = 'regexp' and regexp_like(indexed_val, p_indexed_val))
        )
    
    -- actually we would pass the tf_params above if it were a real table function
    select * from tf_sel
    
    -- >>>
    
    -- ID INDEXED_VAL X
    -- -- ----------- --------
    --  1 ab12        asdfasdf
    
    
    
    -- Question B: How can we speed up data_union with dg_filter to be as fast as the data_group1 query which
    --             actually provides the same result? 
    -- 
    -- A very similar approach is considered in other scenarios where we like to join the results of 
    -- different queries (>5) returning joinable data and beeing filtered based on the same parameters.
    
    -- <<< comment the following lines to simulate the union problem
    
    --  , data_group1 as (  -- may run quite fast
    --      select 'dg1' dg_id, q.* from complex_query q where x < 'a'  -- just an example returning some special rows that should be filtered later on!
    --  )
    --  
    --  , data_group2 as (  -- may run quite fast
    --      select 'dg2' dg_id, q.* from complex_query q where instr(x,'p') >= 0  -- just an example returning some special rows that should be filtered later on!
    --  )   
    --  
    --  
    --  , dg_filter as (  -- may be set by a report or indirectly by user filters
    --      select  'dg1' dg_id  from dual
    --  )
    --  
    --  , data_union as (  -- runs much slower due to another execution plan
    --      select * from (
    --          select * from data_group1 
    --          union all select * from data_group2
    --      )
    --      where dg_id in (select dg_id from dg_filter)
    --  )
    --
    --select * from data_union
    
    -- >>>
    
    -- DG_ID ID INDEXED_VAL X
    -- ----- -- ----------- -------------
    -- dg1    4 xy55         asdja´sf asd
    

    这是对jonearles

    提供的示例代码和答案的评论

    实际上你的回答是我(在某些情况下无关,虽然在一起)用例A和B的混合。虽然你提到优化器具有动态FILTER和其他功能仍然是必不可少的。

    用例B(“数据分区/组联合”)

    实际上,用例B(基于您的示例表)看起来更像是这样,但我仍然需要检查真实场景中的性能问题。也许你已经看到了它的一些问题?

    select * from (
        select 'dg1' data_group, x.* from sample_table x 
            where mod(to_number(some_other_column1), 100000) = 0  -- just some example restriction
                --and indexed_val = '3635'  -- commenting this in and executing this standalone returns:
                ----------------------------------------------------------------------------------------
                --| Id  | Operation                   | Name              | Rows  | Bytes | Cost (%CPU)|
                ----------------------------------------------------------------------------------------
                --|   0 | SELECT STATEMENT            |                   |     1 |    23 |     2   (0)|
                --|   1 |  TABLE ACCESS BY INDEX ROWID| SAMPLE_TABLE      |     1 |    23 |     2   (0)|
                --|   2 |   INDEX RANGE SCAN          | SAMPLE_TABLE_IDX1 |     1 |       |     1   (0)|
                ----------------------------------------------------------------------------------------            
        union all
        select 'dg2', x.* from sample_table x
            where mod(to_number(some_other_column2), 9999) = 0  -- just some example restriction
        union all
        select 'dg3', x.* from sample_table x
            where mod(to_number(some_other_column3), 3635) = 0  -- just some example restriction
    )
    where data_group in ('dg1') and indexed_val = '35'
    
    -------------------------------------------------------------------------------------------
    --| Id  | Operation                      | Name              | Rows  | Bytes | Cost (%CPU)|
    -------------------------------------------------------------------------------------------
    --|   0 | SELECT STATEMENT               |                   |     3 |   639 |     2   (0)|
    --|   1 |  VIEW                          |                   |     3 |   639 |     2   (0)|
    --|   2 |   UNION-ALL                    |                   |       |       |            |
    --|   3 |    TABLE ACCESS BY INDEX ROWID | SAMPLE_TABLE      |     1 |    23 |     2   (0)|
    --|   4 |     INDEX RANGE SCAN           | SAMPLE_TABLE_IDX1 |     1 |       |     1   (0)|
    --|   5 |    FILTER                      |                   |       |       |            |
    --|   6 |     TABLE ACCESS BY INDEX ROWID| SAMPLE_TABLE      |     1 |    23 |     2   (0)|
    --|   7 |      INDEX RANGE SCAN          | SAMPLE_TABLE_IDX1 |     1 |       |     1   (0)|
    --|   8 |    FILTER                      |                   |       |       |            |
    --|   9 |     TABLE ACCESS BY INDEX ROWID| SAMPLE_TABLE      |     1 |    23 |     2   (0)|
    --|  10 |      INDEX RANGE SCAN          | SAMPLE_TABLE_IDX1 |     1 |       |     1   (0)|
    -------------------------------------------------------------------------------------------
    

    用例A(按列查询类型过滤)

    根据您的样本表,这更像我想做的事情。 如您所见,只有快速where p.ft_id = 'eq' and x.indexed_val = p.val查询显示了索引的使用情况,但where子句中包含所有不同的过滤器选项将导致计划切换始终使用全表扫描: - / (即使我在SQL中的任何地方使用:p_filter_type:p_indexed_val_filter而不是在我放置的那个地方,它也不会改变。)

    with 
        filter_type as (
            select 'all' as id from dual
            union all select 'eq' as id from dual
            union all select 'like' as id from dual
            union all select 'regexp' as id from dual
        )
        , params as (
            select 
                (select * from filter_type where id = :p_filter_type) as ft_id,
                :p_indexed_val_filter as val
            from dual
        )
    select * 
    from params p
        join sample_table x on (1=1)
        -- the following with the above would show the 'eq' use case with a fast index scan (plan id 14/15)
        --where p.ft_id = 'eq' and x.indexed_val = p.val
        ------------------------------------------------------------------------------------------
        --| Id  | Operation                     | Name              | Rows  | Bytes | Cost (%CPU)|
        ------------------------------------------------------------------------------------------
        --|   0 | SELECT STATEMENT              |                   |     1 |    23 |    12   (0)|
        --|   1 |  VIEW                         |                   |     4 |    20 |     8   (0)|
        --|   2 |   UNION-ALL                   |                   |       |       |            |
        --|   3 |    FILTER                     |                   |       |       |            |
        --|   4 |     FAST DUAL                 |                   |     1 |       |     2   (0)|
        --|   5 |    FILTER                     |                   |       |       |            |
        --|   6 |     FAST DUAL                 |                   |     1 |       |     2   (0)|
        --|   7 |    FILTER                     |                   |       |       |            |
        --|   8 |     FAST DUAL                 |                   |     1 |       |     2   (0)|
        --|   9 |    FILTER                     |                   |       |       |            |
        --|  10 |     FAST DUAL                 |                   |     1 |       |     2   (0)|
        --|  11 |  FILTER                       |                   |       |       |            |
        --|  12 |   NESTED LOOPS                |                   |     1 |    23 |     4   (0)|
        --|  13 |    FAST DUAL                  |                   |     1 |       |     2   (0)|
        --|  14 |    TABLE ACCESS BY INDEX ROWID| SAMPLE_TABLE      |     1 |    23 |     2   (0)|
        --|  15 |     INDEX RANGE SCAN          | SAMPLE_TABLE_IDX1 |     1 |       |     1   (0)|
        --|  16 |   VIEW                        |                   |     4 |    20 |     8   (0)|
        --|  17 |    UNION-ALL                  |                   |       |       |            |
        --|  18 |     FILTER                    |                   |       |       |            |
        --|  19 |      FAST DUAL                |                   |     1 |       |     2   (0)|
        --|  20 |     FILTER                    |                   |       |       |            |
        --|  21 |      FAST DUAL                |                   |     1 |       |     2   (0)|
        --|  22 |     FILTER                    |                   |       |       |            |
        --|  23 |      FAST DUAL                |                   |     1 |       |     2   (0)|
        --|  24 |     FILTER                    |                   |       |       |            |
        --|  25 |      FAST DUAL                |                   |     1 |       |     2   (0)|
        ------------------------------------------------------------------------------------------  
    where 
        --mod(to_number(some_other_column1), 3000) = 0 and  -- just some example restriction
        (
                p.ft_id = 'all'
            or
                p.ft_id = 'eq' and x.indexed_val = p.val
            or 
                p.ft_id = 'like' and x.indexed_val like p.val
            or 
                p.ft_id = 'regexp' and regexp_like(x.indexed_val, p.val)
        )
    -- with the full flexibility of the filter the plan shows a full table scan (plan id 13) :-(    
    --------------------------------------------------------------------------
    --| Id  | Operation          | Name         | Rows  | Bytes | Cost (%CPU)|
    --------------------------------------------------------------------------
    --|   0 | SELECT STATEMENT   |              |  1099 | 25277 |   115   (3)|
    --|   1 |  VIEW              |              |     4 |    20 |     8   (0)|
    --|   2 |   UNION-ALL        |              |       |       |            |
    --|   3 |    FILTER          |              |       |       |            |
    --|   4 |     FAST DUAL      |              |     1 |       |     2   (0)|
    --|   5 |    FILTER          |              |       |       |            |
    --|   6 |     FAST DUAL      |              |     1 |       |     2   (0)|
    --|   7 |    FILTER          |              |       |       |            |
    --|   8 |     FAST DUAL      |              |     1 |       |     2   (0)|
    --|   9 |    FILTER          |              |       |       |            |
    --|  10 |     FAST DUAL      |              |     1 |       |     2   (0)|
    --|  11 |  NESTED LOOPS      |              |  1099 | 25277 |   115   (3)|
    --|  12 |   FAST DUAL        |              |     1 |       |     2   (0)|
    --|  13 |   TABLE ACCESS FULL| SAMPLE_TABLE |  1099 | 25277 |   113   (3)|
    --|  14 |    VIEW            |              |     4 |    20 |     8   (0)|
    --|  15 |     UNION-ALL      |              |       |       |            |
    --|  16 |      FILTER        |              |       |       |            |
    --|  17 |       FAST DUAL    |              |     1 |       |     2   (0)|
    --|  18 |      FILTER        |              |       |       |            |
    --|  19 |       FAST DUAL    |              |     1 |       |     2   (0)|
    --|  20 |      FILTER        |              |       |       |            |
    --|  21 |       FAST DUAL    |              |     1 |       |     2   (0)|
    --|  22 |      FILTER        |              |       |       |            |
    --|  23 |       FAST DUAL    |              |     1 |       |     2   (0)|
    --------------------------------------------------------------------------
    

2 个答案:

答案 0 :(得分:1)

多个功能使优化器能够生成动态计划。最常见的功能是FILTER操作,不应与过滤谓词混淆。 FILTER操作允许Oracle在运行时根据动态值启用或禁用部分计划。此功能通常适用于绑定变量,其他类型的动态查询可能不使用它。

示例架构

create table sample_table
(
    indexed_val        varchar2(100),
    some_other_column1 varchar2(100),
    some_other_column2 varchar2(100),
    some_other_column3 varchar2(100)
);

insert into sample_table
select level, level, level, level
from dual
connect by level <= 100000;

create index sample_table_idx1 on sample_table(indexed_val);

begin
    dbms_stats.gather_table_stats(user, 'sample_table');
end;
/

使用绑定变量的示例查询

explain plan for
select * from sample_table where :p_filter_type = 'all'
union all
select * from sample_table where :p_filter_type = 'eq'     and indexed_val = :p_indexed_val
union all
select * from sample_table where :p_filter_type = 'like'   and indexed_val like :p_indexed_val
union all
select * from sample_table where :p_filter_type = 'regexp' and regexp_like(indexed_val, :p_indexed_val);

select * from table(dbms_xplan.display(format => '-cost -bytes -rows'));

示例计划

这表明根据输入使用了截然不同的计划。单个=将使用INDEX RANGE SCAN,没有谓词将使用TABLE ACCESS FULL。该  正则表达式也使用全表扫描,因为无法索引正则表达式。虽然取决于它可能的确切表达类型  可以通过基于函数的索引或Oracle Text索引启用有用的索引。

Plan hash value: 100704550

------------------------------------------------------------------------------
| Id  | Operation                             | Name              | Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                      |                   | 00:00:01 |
|   1 |  UNION-ALL                            |                   |          |
|*  2 |   FILTER                              |                   |          |
|   3 |    TABLE ACCESS FULL                  | SAMPLE_TABLE      | 00:00:01 |
|*  4 |   FILTER                              |                   |          |
|   5 |    TABLE ACCESS BY INDEX ROWID BATCHED| SAMPLE_TABLE      | 00:00:01 |
|*  6 |     INDEX RANGE SCAN                  | SAMPLE_TABLE_IDX1 | 00:00:01 |
|*  7 |   FILTER                              |                   |          |
|   8 |    TABLE ACCESS BY INDEX ROWID BATCHED| SAMPLE_TABLE      | 00:00:01 |
|*  9 |     INDEX RANGE SCAN                  | SAMPLE_TABLE_IDX1 | 00:00:01 |
|* 10 |   FILTER                              |                   |          |
|* 11 |    TABLE ACCESS FULL                  | SAMPLE_TABLE      | 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter(:P_FILTER_TYPE='all')
   4 - filter(:P_FILTER_TYPE='eq')
   6 - access("INDEXED_VAL"=:P_INDEXED_VAL)
   7 - filter(:P_FILTER_TYPE='like')
   9 - access("INDEXED_VAL" LIKE :P_INDEXED_VAL)
       filter("INDEXED_VAL" LIKE :P_INDEXED_VAL)
  10 - filter(:P_FILTER_TYPE='regexp')
  11 - filter( REGEXP_LIKE ("INDEXED_VAL",:P_INDEXED_VAL))

答案 1 :(得分:0)

(更多情况 A)但也适用于 B)以这种方式......)

我现在正在使用一些混合方法(我的问题中的1.和2.点的组合)并且实际上非常喜欢它,因为它还提供了良好的调试和封装可能性< / strong>并且优化器根本不需要根据更大查询中基本上逻辑上分离的查询找到最佳策略,例如关于内部FILTER规则,这些规则可能是好的,或者最糟糕的是效率低得多:

  1. 在报告中使用此功能

    select * 
    from table(my_report_data_func_sql(
      :val1,
      :val1_filter_type,
      :val2
    ))
    
  2. 表函数的定义如下

    create or replace function my_report_data_func_sql(
      p_val1              integer   default 1234,
      p_val1_filter_type  varchar2  default 'eq',
      p_val2              varchar2  default null
    ) return varchar2 is
    
      query varchar2(4000) := ' 
        with params as (  -- *: default param
            select
                ''||p_val1||'' p_val1,                            -- eq*
                '''||p_val1_filter_type||''' p_val1_filter_type,  -- [eq, all*, like, regexp]
                '''||p_val2||''' p_val2                           -- null*
            from dual
        )
        select x.*
        from
            params p  -- workaround for standalone-sql-debugging using "with" statement above
            join my_report_data_base_view x on (1=1)
        where 1=1  -- ease of filter expression adding below
        '          
        -- #### FILTER CRITERIAS are appended here ####
    
        -- val1-filter
        ||case p_val1_filter_type
            when 'eq' then '
            and val1 = p_val1
        '   when 'like' then '
            and val1 like p_val1
        '   when 'regexp' then '
            and regexp_like(val1,  p_val1)
        '   else '' end  -- all
    
      ;
    begin
      return query;
    end;
    ;
    
  3. 并将通过示例生成以下内容:

    select * 
    from table(my_report_data_func_sql(
      1234,
      'eq',
      'someval2'
    ))
    
    /*
            with params as (  -- *: default param
                select
                    1          p_val1,              -- eq*
                    'eq'       p_val1_filter_type,  -- [eq, all*, like, regexp]
                    'someval2' p_val2               -- null*
                from dual
            )
            select x.*
            from
                params p  -- workaround for standalone-sql-debugging using "with" statement above
                join my_report_data_base_view x on (1=1)
            where 1=1  -- ease of filter expression adding below
                and val1 = p_val1
    */