Question

我们使用python脚本来处理数据，使用TRANSFORM将它们与hive一起使用。我做了两张相同的桌子。首先将数据存储在ORC中，第二个存储在TEXT中。存储在ORC中的数据映射器运行x10比存储在TEXT中的数据慢。

1）我们使用默认的ORC设置。

2）查询：

use db1;

set hive.vectorized.execution.enabled=false;
set hive.vectorized.execution.reduce.enabled=false;
set hive.execution.engine=tez;

ADD FILE /home/user/manage.py;
ADD FILE /home/user/cluster.mod;
ADD FILE /home/user/hive_serde.so;

ALTER TABLE aggregates ADD IF NOT EXISTS PARTITION (dt='${hiveconf:HIVE_CALC_DT}');

FROM (
  FROM visits
  MAP logins, geo_id, ua, os, device, version, splits, entrance_utm,
  is_bounce_visit, is_new_visit, screen, browser, um, ts_start, ts_end,
  ruid, project_id, geo_parents, custom_vars, page_views, totals
  ROW FORMAT DELIMITED
  USING 'python2.7 manage.py metrics_from_visits'
  AS project_id, splits, message_type, message
  WHERE dt = '${hiveconf:HIVE_CALC_DT}'
  CLUSTER BY project_id, splits, message_type) map_output

INSERT OVERWRITE TABLE aggregates partition (dt='${hiveconf:HIVE_CALC_DT}')

REDUCE *
  USING 'python2.7 manage.py generate_aggregates --date ${hiveconf:HIVE_CALC_DT}'
  AS (
    table_name string COMMENT 'Aggregator type',
    key string COMMENT 'Key for Hbase',
    column_family string COMMENT 'Column family name for Hbase',
    column_name string COMMENT 'Column name for Hbase',
    data string COMMENT 'Json data from metric aggregates'
  )

ROW FORMAT DELIMITED;

任何可能发生这种情况的想法都会有很大帮助。

使用ORC文件，Hive转换查询运行缓慢

0 个答案: