我正在尝试将xml文件加载到hive表中。我正在使用xml serde [here] [1]。我能够加载简单的平面xml文件。但是当xml中有嵌套元素时,我使用hive复杂数据类型来存储它们(例如,array<struct>
)。下面是我尝试加载的示例xml。我的目标是将所有元素,属性和内容加载到hive表中。
<classif action="del">
<code>123</code>
<class action="aou">
<party>p1</party>
<description action="up">
<name action="aorup" ln="te">
this is name1
</name>
<name action="aorup" ln="tm">
this is name2
</name>
<name action="aorup" ln="hi">
this is name2
</name>
</description>
</class>
<class action="a">
<party>p2</party>
<description action="up">
<name action="aorup" ln="te">
this is name4
</name>
<name action="aorup" ln="tm">
this is name5
</name>
<name action="aorup" ln="hi">
this is name6
</name>
</description>
</class>
</classif>
我想要获得的Hive输出是......
{action:"del", classif:{code:"123", class:[{action:"aou", class:{party:"p1", description:{action:"up", description:[{action:"aorup", ln:"te", name:"this is name1"}, {action:"aorup", ln:"tm", name:"this is name2"}, {action:"aorup", ln:"hi", name:"this is name3"}]}}}, {action:"a", class:{party:"p2", description:{action:"up", description:[{action:"aorup", ln:"te", name:"this is name4"}, {action:"aorup", ln:"tm", name:"this is name5"}, {action:"aorup", ln:"hi", name:"this is name6"}]}}}]}}
我想将整个xml加载到一个hive列中。我尝试了以下方法:
DROP TABLE classif;
CREATE TABLE classif(
classif STRUCT<
Action:STRING, classif:STRUCT<Code:STRING, class:ARRAY<STRUCT<Action:STRING, class:STRUCT<party:STRING, description:STRUCT<action:STRING,description:ARRAY<STRUCT<action:STRING,ln:STRING,name:STRING>>>
>>>
>>)
ROW FORMAT SERDE 'com.ibm.spss.hive.serde2.xml.XmlSerDe'
WITH SERDEPROPERTIES (
"xml.processor.class"="com.ximpleware.hive.serde2.xml.vtd.XmlProcessor",
"column.xpath.classif"="/classif")
STORED AS INPUTFORMAT 'com.ibm.spss.hive.serde2.xml.XmlInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
TBLPROPERTIES ("xmlinput.start"="<classif ","xmlinput.end"= "</classif>");
我得到的输出:
{"action":"del","classif":{"code":"123","class":[{"action":null,"class":null},{"action":"up","class":null},{"action":null,"class":null},{"action":"up","class":null}]}}