Question

我正在尝试将以下Json数据加载到我的HDP中的Pig中。

$ cat myfile1.json

[
    {
        color: "red",
        value: "#f00"
    },
    {
        color: "green",
        value: "#0f0"
    },
    {
        color: "blue",
        value: "#00f"
    },
    {
        color: "cyan",
        value: "#0ff"
    },
    {
        color: "magenta",
        value: "#f0f"
    },
    {
        color: "yellow",
        value: "#ff0"
    },
    {
        color: "black",
        value: "#000"
    }
]

我尝试了下面的Pig声明并获得了ERROR。我无法找到错误。我应该在JsonLoader中提到的正确格式是什么。

a = load '/root/myfile1.json' using JsonLoader('{(color:chararray,value:chararray)}');

dump a;

java.lang.Exception: org.codehaus.jackson.JsonParseException: Unexpected end-of-input within/between OBJECT entries
 at [Source: java.io.ByteArrayInputStream@27c85f78; line: 1, column: 35]
        at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: org.codehaus.jackson.JsonParseException: Unexpected end-of-input within/between OBJECT entries
 at [Source: java.io.ByteArrayInputStream@27c85f78; line: 1, column: 35]
        at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1291)
        at org.codehaus.jackson.impl.Utf8StreamParser._skipWS(Utf8StreamParser.java:1817)
        at org.codehaus.jackson.impl.Utf8StreamParser.nextToken(Utf8StreamParser.java:315)
        at org.apache.pig.builtin.JsonLoader.getNext(JsonLoader.java:167)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.nextKeyValue(PigRecordReader.java:211)
        at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:533)
        at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
        at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
2016-01-19 03:59:27,053 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2016-01-19 03:59:27,053 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local1639714432_0007 has failed! Stop running all dependent jobs
2016-01-19 03:59:27,053 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-01-19 03:59:27,054 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2016-01-19 03:59:27,054 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats reported below may be incomplete
2016-01-19 03:59:27,055 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:

Answer 1

我认为您的第一行代码应如下所示

a = load '/root/myfile1.json' using JsonLoader('color:chararray,value:chararray');

有关示例，请参阅此link。

更新添加了JSON数据文件的压缩版本

[
 {color: "red",value: "#f00"},
 {color: "green",value: "#0f0"},
 {color: "blue",value: "#00f"},
 {color: "cyan",value: "#0ff"},
 {color: "magenta",value: "#f0f"},
 {color: "yellow",value: "#ff0"},
 {color: "black",value: "#000"}
]

希望这有帮助。

在Apache Pig中加载Json数据时出错

1 个答案: