Question

我对Hive和ElasticMapreduce相当新，目前我遇到了一个特定的问题。在具有数十亿行JSON对象的表上运行Hive语句时，只要其中一行是无效/格式错误的JSON，MapReduce作业就会崩溃。

例外：

java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"ip":"39488130","cdate":"2012-08-09","cdate_ts":"2012-08-09 17:06:41","country":"SA","city":"Riyadh","mid":"6666276582211270592","osversion":"5.1.
1
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:441)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:377)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1132)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing writable {"ip":"39488130","cdate":"2012-08-09","cdate_ts":"2012-08-09 17:06:41","country":"SA","city":"Riyadh","mid":"6666276582211270592","osversion":"5.1.1
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:524)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
... 8 more
Caused by: com.google.gson.JsonSyntaxException: com.google.gson.stream.MalformedJsonException: Unterminated string near
at com.google.gson.Streams.parse(Streams.java:51)
at com.google.gson.JsonParser.parse(JsonParser.java:83)
at com.google.gson.JsonParser.parse(JsonParser.java:58)
at com.google.gson.JsonParser.parse(JsonParser.java:44)
at com.amazon.elasticmapreduce.JsonSerde.deserialize(Unknown Source)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:510)
... 9 more
Caused by: com.google.gson.stream.MalformedJsonException: Unterminated string near
at com.google.gson.stream.JsonReader.syntaxError(JsonReader.java:1110)
at com.google.gson.stream.JsonReader.nextString(JsonReader.java:967)
at com.google.gson.stream.JsonReader.nextValue(JsonReader.java:802)
at com.google.gson.stream.JsonReader.objectValue(JsonReader.java:782)
at com.google.gson.stream.JsonReader.quickPeek(JsonReader.java:377)
at com.google.gson.stream.JsonReader.peek(JsonReader.java:340)
at com.google.gson.Streams.parseRecursive(Streams.java:60)
at com.google.gson.Streams.parseRecursive(Streams.java:83)
at com.google.gson.Streams.parse(Streams.java:40)
... 14 more

我按照以下方式创建表格：

CREATE EXTERNAL TABLE IF NOT EXISTS table1 (
column1 string,
column2 string
)
PARTITIONED BY (year string, month string)
ROW FORMAT SERDE 'com.amazon.elasticmapreduce.JsonSerde'
WITH SERDEPROPERTIES ('paths'='c1, c2')
LOCATION 's3://mybucket/table1';

我该怎么做才能防止崩溃？忽略格式错误的JSON对象/字符串就可以了，因为它有十分之一的格式错误。

提前感谢您的帮助。最好的，Sascha

Answer 1

通过更改行格式中使用的类并添加＆＃39;格式错误的＆＃39;属性，您可以使表格创建处理格式错误的JSON：

ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES ("ignore.malformed.json" = "true")
LOCATION ...

使用＆＃39; hive.aux.jars.path＆＃39;包含JAR。属于＆＃39; hive-site.xml＆＃39;或者加入JAR＆＃39;蜂巢指令。您可以找到JAR here，或者从this source编译它。

Answer 2

来自Apache的JsonSerDe似乎忽略了格式错误的JSON字符串...... http://code.google.com/p/hive-json-serde/

Answer 3

基本上，由于JSON无效字符串，因此发生上述错误。尝试解决此问题。

最明智的是，为了避免崩溃您的应用，请在try catch块和进程中捕获该异常。这样您的应用就不会崩溃。

Hive / ElasticMapreduce：如何使JsonSerDe忽略格式错误的JSON？

3 个答案: