Question

我只是尝试执行这个愚蠢的脚本：

a = LOAD 'example';
dump a ;

其中'example'是hdfs主页中的文件，我得到了这个例外：

ERROR 2017: Internal error creating job configuration.

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias a
        at org.apache.pig.PigServer.openIterator(PigServer.java:836)
        at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696)
        at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
        at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
        at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
        at org.apache.pig.Main.run(Main.java:538)
        at org.apache.pig.Main.main(Main.java:157)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156) Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias a
        at org.apache.pig.PigServer.storeEx(PigServer.java:935)
        at org.apache.pig.PigServer.store(PigServer.java:898)
        at org.apache.pig.PigServer.openIterator(PigServer.java:811)
        ... 12 more Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: ERROR 2017: Internal error creating job configuration.
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
        at org.apache.pig.PigServer.launchPlan(PigServer.java:1264)
        at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1249)
        at org.apache.pig.PigServer.storeEx(PigServer.java:931)
        ... 14 more Caused by: java.io.NotSerializableException: org.apache.log4j.Level
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
        at org.apache.pig.impl.util.JarManager.createJar(JarManager.java:186)
        at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:517)
        ... 19 more

第一个原因是“java.io.NotSerializableException：org.apache.log4j.Level”。

我用Pig 0.11.0,0.11.1和0.12.1进行了测试。

Java版“1.6.0_31”

有什么想法吗？

Answer 1

我在路径中有Hadoop：

PATH=/home/.../hadoop/bin:...

并且Pig不仅读取配置文件~/hadoop/conf/core-site.xml和~/hadoop/conf/mapred-site.xml，还检查~/hadoop/lib/*，并且似乎是该目录中的jar和Pig安装的jar之间的一些交互。 / p>

我从~/hadoop/bin删除了PATH，将配置文件core-site.xml和mapred-site.xml复制到~/pig/conf，一切都开始有效。

Simple Pig脚本抛出NotSerializableException：org.apache.log4j.Level

1 个答案: