Question

我正在MRv2上使用Hive。我准备了一个包含十亿个元组的表。我写了类似EXPLAIN select count(*) from table的查询。 EXPLAIN结果如下：

1> STAGE DEPENDENCIES:   Stage-1 is a root stage   Stage-0 depends on
2> stages: Stage-1
3> 
4> STAGE PLANS:   Stage: Stage-1
5>     Map Reduce
6>       Map Operator Tree:
7>           TableScan
8>             alias: table
9>             Statistics: Num rows: 1 Data size: 39328468992 Basic stats: COMPLETE Column stats: COMPLETE
10>             Select Operator
11>               Statistics: Num rows: 1 Data size: 39328468992 Basic stats: COMPLETE Column stats: COMPLETE
13>               Group By Operator
14>                 aggregations: count()
15>                 mode: hash
16>                 outputColumnNames: _col0
17>                 Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
18>                 Reduce Output Operator
19>                   sort order: 
20>                   Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
21>                   value expressions: _col0 (type: bigint)
22>       Reduce Operator Tree:
23>         Group By Operator
24>           aggregations: count(VALUE._col0)
25>           mode: mergepartial
26>           outputColumnNames: _col0
27>           Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
28>           File Output Operator
29>             compressed: false
30>             Statistics: Num rows: 1 Data size: 8 Basic stats: COMPLETE Column stats: COMPLETE
31>             table:
32>                 input format: org.apache.hadoop.mapred.TextInputFormat
33>                 output format: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
34>                 serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
35> 
36>   Stage: Stage-0
37>     Fetch Operator
38>       limit: -1
39>       Processor Tree:
40>         ListSink

在第9行的统计信息中，尽管此表有十亿个元组，但是Num rows是1。

请教我如何计算EXPLAIN中的统计结果。

Answer 1

可能是外部表。由于Hive不了解元数据，因此它只显示一行。

例如：

Map Operator Tree:
      TableScan
        alias: test_ext_tb
        Statistics: **Num rows: 1 Data size: 452001824** Basic stats: COMPLETE Column stats: COMPLETE
        Select Operator
          Statistics: Num rows: 1 Data size: 452001824 Basic stats: COMPLETE Column stats: COMPLETE
          Group By Operator
            aggregations: count()

蜂巢：无法理解EXAMPLE结果中的统计信息

1 个答案: