Hive代码卡在reduce = 99%

时间:2016-09-06 09:57:34

标签: hadoop hive hiveql skew

我运行了一段hive代码。但它坚持了99%。 这是我的代码:

    insert overwrite table novaya.brand_info 
select goods.brandid, goods.brandnm, cate1.cate1_price, cate2.cate2_price, cate3.cate3_price
  from novaya.goods_info goods 
    left join 
    (select brandid, brandnm, mng_catecode1, avg(sales_price) as cate1_price
    from novaya.goods_info  where brandid is not null 
    group by brandid, brandnm, mng_catecode1) cate1 
    on goods.brandid=cate1.brandid
    left join 
    (select brandid, brandnm, mng_catecode2, avg(sales_price) as cate2_price
    from novaya.goods_info  where brandid is not null 
    group by brandid, brandnm, mng_catecode2) cate2
    on  goods.brandid=cate2.brandid
    left join 
    (select brandid, brandnm, mng_catecode3, avg(sales_price) as cate3_price
    from novaya.goods_info  where brandid is not null 
    group by brandid, brandnm, mng_catecode3) cate3  
    on goods.brandid=cate3.brandid
  where goods.brandid is not null ;

cate1,cate2和cate3将属于商品表brandid和brandnm之后的列,这个逻辑并不复杂。商品表可能比三个表更大。 它可能由歪斜引起。如何在不更改配置单元设置的情况下解决此问题。 唔谢谢。

0 个答案:

没有答案