Hive - 插入表分区抛出错误

时间:2018-03-29 05:27:54

标签: apache-spark hive hiveql spark-thriftserver

我正在尝试在Hive上在spark上创建一个分区表,并使用Hive中其他表中的数据加载它。 加载数据时出现以下错误:

  

错误:org.apache.spark.sql.AnalysisException:   org.apache.hadoop.hive.ql.metadata.Table.ValidationFailureSemanticException:   分区规范{cardsuit =,cardcolor =,cardSuit = SPA,cardColor = BLA}   包含非分区列;

以下是用于执行任务的命令: -

create table if not exists hive_tutorial.hive_table(color string, suit string,value string) comment 'first hive table' row format delimited fields terminated by '|' stored as TEXTFILE;

LOAD DATA LOCAL INPATH 'file:///E:/Kapil/software-study/Big_Data_NoSql/hive/deckofcards.txt' OVERWRITE INTO TABLE hive_table; --data is correctly populated(52 rows)

SET hive.exec.dynamic.partition = true;

SET hive.exec.dynamic.partition.mode = nonstrict;

create table if not exists hive_tutorial.hive_table_partitioned(color string, suit string,value int) comment 'first hive table' partitioned by (cardSuit string,cardColor string) row format delimited fields terminated by '|' stored as TEXTFILE;

INSERT INTO TABLE hive_table_partitioned PARTITION (cardSuit,cardColor) select color,suit,value,substr(suit, 1, 3) as cardSuit,substr(color, 1, 3) as cardColor from hive_table;

--alternatively i tried
INSERT OVERWRITE  TABLE hive_table_partitioned PARTITION (cardSuit,cardColor) select color,suit,value,substr(suit, 1, 3) as cardSuit,substr(color, 1, 3) as cardColor from hive_table;

数据样本: -

BLACK | SPADE | 2

BLACK | SPADE | 3

BLACK | SPADE | 4

BLACK | SPADE | 5

BLACK | SPADE | 6

BLACK | SPADE | 7

BLACK | SPADE | 8

BLACK | SPADE | 9

我正在使用spark 2.2.0和java版本1.8.0_31。

我已经检查并尝试过类似线程中给出的答案,但无法解决我的问题: - SemanticException Partition spec {col=null} contains non-partition columns

我在这里错过了什么吗?

1 个答案:

答案 0 :(得分:0)

在创建表时仔细阅读上面的脚本后,value列是分区表中的int类型,其中为原始字符串。错误误导了!!!