Question

我正在尝试基于员工记录的部门执行hive静态分区，当我执行load命令将数据插入到我的分区表中时，它会根据我在load data命令中指定的分区值更改所有dept值。

我在hdfs中的数据如下所示：

1,adi,Admin,50000,A
2,Gokul,Admin,50000,B
3,Janet,Sales,60000,A
4,Hari,Admin,50000,C
5,Sanker,Admin,50000,C
6,Margaret,Tech,12000,A
7,Nirmal,Tech,12000,B
8,jinju,Engineer,45000,B
9,Nancy,Admin,50000,A
10,Andrew,Manager,40000,A
11,Arun,Manager,40000,B
12,Harish,Sales,60000,B
13,Robert,Manager,40000,A
14,Laura,Engineer,45000,A
15,Anju,Ceo,100000,B
16,Aarathi,Manager,40000,B
17,Parvathy,Engineer,45000,B
18,Gopika,Admin,50000,B
19,Steven,Engineer,45000,A
20,Michael,Ceo,100000,A

我的分区表如下所示：

create table employee(
id string,
name string,
role string,
salary string)
partitioned by (dept string)
row format delimited
fields terminated by ','
lines terminated by '\n'
stored as textfile;

load command
load data inpath '/user/adithyan/employee.txt' overwrite into table employee partition (dept='A');

我已经在上面给出了输入数据，在执行之后，这个命令记录已经从hdfs插入到hive，但它加载了由＆＃39; A＆＃39;

更改的部门的所有记录

输出

employee.id employee.name   employee.degree employee.salary employee.dept
1   adi Admin   50000   A
2   Gokul   Admin   50000   A
3   Janet   Sales   60000   A
4   Hari    Admin   50000   A
5   Sanker  Admin   50000   A
6   Margaret    Tech    12000   A
7   Nirmal  Tech    12000   A
8   jinju   Engineer    45000   A
9   Nancy   Admin   50000   A
10  Andrew  Manager 40000   A
11  Arun    Manager 40000   A
12  Harish  Sales   60000   A
13  Robert  Manager 40000   A
14  Laura   Engineer    45000   A
15  Anju    Ceo 100000  A
16  Aarathi Manager 40000   A
17  Parvathy    Engineer    45000   A
18  Gopika  Admin   50000   A
19  Steven  Engineer    45000   A
20  Michael Ceo 100000  A

所有部门都已更改为A，这是错误的。有人可以帮我解决如何将分区数据插入表格的问题吗？

Answer 1

似乎加载数据不适合该作业。

加载操作目前是移动的纯复制/移动操作   数据文件到与Hive表对应的位置。

https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables

但是，您可以使用外部表来实现目标。

create external table employee_ext
(
    id      string
   ,name    string
   ,role    string
   ,salary  string
   ,dept    string
)
row format delimited
fields terminated by ','
lines terminated by '\n'
stored as textfile
location '/user/adithyan/'
;

insert into employee partition (dept) select * from employee_ext;

select * from employee
;

+-------------+---------------+---------------+-----------------+---------------+
| employee.id | employee.name | employee.role | employee.salary | employee.dept |
+-------------+---------------+---------------+-----------------+---------------+
|           1 | adi           | Admin         |           50000 | A             |
|           3 | Janet         | Sales         |           60000 | A             |
|           6 | Margaret      | Tech          |           12000 | A             |
|           9 | Nancy         | Admin         |           50000 | A             |
|          10 | Andrew        | Manager       |           40000 | A             |
|          13 | Robert        | Manager       |           40000 | A             |
|          14 | Laura         | Engineer      |           45000 | A             |
|          19 | Steven        | Engineer      |           45000 | A             |
|          20 | Michael       | Ceo           |          100000 | A             |
|           2 | Gokul         | Admin         |           50000 | B             |
|           7 | Nirmal        | Tech          |           12000 | B             |
|           8 | jinju         | Engineer      |           45000 | B             |
|          11 | Arun          | Manager       |           40000 | B             |
|          12 | Harish        | Sales         |           60000 | B             |
|          15 | Anju          | Ceo           |          100000 | B             |
|          16 | Aarathi       | Manager       |           40000 | B             |
|          17 | Parvathy      | Engineer      |           45000 | B             |
|          18 | Gopika        | Admin         |           50000 | B             |
|           4 | Hari          | Admin         |           50000 | C             |
|           5 | Sanker        | Admin         |           50000 | C             |
+-------------+---------------+---------------+-----------------+---------------+

Hive静态分区加载所有记录

1 个答案: