根据条件动态帧过滤条件

时间:2019-12-20 04:13:48

标签: sql apache-spark hive apache-spark-sql hiveql

我们为条件/过滤器保留了单独的表格。根据条件,在基表上应用过滤器。

这是示例输入条件数据,仅供参考

+-----------+-------------------+----------+------------+--------------+---------------+----------------+
val_range   |val_range_operator | val_From |  val_till  |   val_except  |  except_from |   except_till  |
+-----------+-------------------+----------+------------+---------------+--------------+----------------+
            |                   |          |            |               |              |                |
100         |  =                |          |            |               |              |                |
            |                   |          |            |               |              |                |   
            | between           |   200    |    300     |               |              |                |
            |                   |          |            |               |              |                |
            | between           |   410    |    620     | 450,600,610   |              |                |
            |                   |          |            |               |              |                |
            | between           |   800    |   999      | 810,820,850   |     890      |     930        |
            |                   |          |            |               |              |                |
            | between           |   1200   |   1500     |               |     1300     |      1399      |
+-----------+-------------------+----------+------------+---------------+--------------+----------------+

根据此输入条件,过滤器的推导如下。

Select col* 
from transaction_tbl 
where 
    val_range = 100 
    OR val_range between 200 AND 300 
    OR val_range between 410 AND 620 AND val_range not in (450,600,610) 
    OR val_range between 800 AND 999 AND val_range not in (810,820,850) AND NOT BETWEEN 890 and 930 
    OR val_range between 1200 AND 1500 AND val_range NOT BETWEEN 1300 AND 1399 

请帮助我实现过滤器查询

2 个答案:

答案 0 :(得分:1)

下面的sparkSQL将帮助您构建where子句

select 
  concat(
    '( ', 
    concat_ws(
      ') OR (', 
      collect_list(
        case when val_range_operator = '=' 
        and val_range is not null then concat_ws(' ', 'val_range', '=', val_range) when val_range_operator = 'between' 
        and val_From is not null 
        and val_till is not null 
        and val_range is null 
        and val_except is null 
        and except_from is null 
        and except_till is null then concat_ws(
          ' ', 'val_range', 'between', val_From, 
          'AND', val_till
        ) when val_range_operator = 'between' 
        and val_From is not null 
        and val_till is not null 
        and val_range is null 
        and val_except is not null 
        and except_from is null 
        and except_till is null then concat_ws(
          ' ', 'val_range', 'between', val_From, 
          'AND', val_till, 'AND', 'val_range', 
          'NOT', 'IN', '(', val_except, ')'
        ) when val_range_operator = 'between' 
        and val_From is not null 
        and val_till is not null 
        and val_range is null 
        and val_except is not null 
        and except_from is not null 
        and except_till is not null then concat_ws(
          ' ', 'val_range', 'between', val_From, 
          'AND', val_till, 'AND', 'val_range', 
          'NOT', 'IN', '(', val_except, ')', 
          'AND NOT BETWEEN', except_from, 
          'AND', except_till
        ) when val_range_operator = 'between' 
        and val_From is not null 
        and val_till is not null 
        and val_range is null 
        and val_except is null 
        and except_from is not null 
        and except_till is not null then concat_ws(
          ' ', 'val_range', 'between', val_From, 
          'AND', val_till, 'AND NOT BETWEEN', 
          except_from, 'AND', except_till
        ) end
      )
    ), 
    ' )'
  ) as filter_condition 
from 
  filter_tb

PS:场景是根据参考数据得出的。如果存在其他任何情况,请更新查询。

答案 1 :(得分:0)

您可以根据输入使用StringBuilder在运行时形成查询,也可以使用QueryDSL之类的东西在运行时形成查询。

http://www.querydsl.com/

相关问题