Question

我遇到了从Hadoop命令行执行PIG命令的问题。命令和错误堆栈位于

之下

我的导师怀疑这是因为HADDOP_HOME和PIG_CLASSPATH不正确。我在HADOOP version 2.8.0。

所以，最初我有HADOOP_HOME作为

HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/

然后我切换了以下设置：

HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/libexec/etc/hadoop

PIG_CLASSPATH定义为$HADOOP_HOME

我在猪身上使用的命令：

A = LOAD '/Users/anarinsky/Downloads/loaddata1.txt';

B = MAPREDUCE '/Users/anarinsky/workspace/wordcount/target/wordcount-1.jar' STORE A INTO '/Users/anarinsky/Downloads/tempwrite2' LOAD  '/Users/anarinsky/Downloads/tempwrite2' AS (word:chararray, count:int) `com.systemskills.hadoop.wordcount.WordCountDriver /wordcountdata /Users/anarinsky/Downloads/pigoptdir`;

猪堆痕迹

ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias B
    at org.apache.pig.PigServer.openIterator(PigServer.java:1019)
    at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:747)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
    at org.apache.pig.Main.run(Main.java:564)
    at org.apache.pig.Main.main(Main.java:176)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias B
    at org.apache.pig.PigServer.storeEx(PigServer.java:1122)
    at org.apache.pig.PigServer.store(PigServer.java:1081)
    at org.apache.pig.PigServer.openIterator(PigServer.java:994)
    ... 13 more
Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:321)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:629)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:308)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1474)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1459)
    at org.apache.pig.PigServer.storeEx(PigServer.java:1118)
    ... 15 more

Answer 1

亚历克斯！不幸的是，它与Pig路径（在我配置的hadoop集群上尝试过）无关，结果相同。您得到的错误是指物理规划编译器在编译方法中存在错误。因此，为了使您的尝试工作，您有两种可能性

使用hadoop运行本机MR作业，并在完成处理后将结果发送到猪

编辑pig源代码并编译自己的版本。你需要编辑 org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler#compile方法并替换

for (PhysicalOperator op : leaves) {
    if (!(op instanceof POStore)) {
        int errCode = 2025;
        String msg = "Expected leaf of reduce plan to " +
            "always be POStore. Found " + op.getClass().getSimpleName();
        throw new MRCompilerException(msg, errCode, PigException.BUG);
    }
}

带

    for (PhysicalOperator op : leaves) {
        if (!(op instanceof POStore) && !(op instanceof PONative)) {
            int errCode = 2025;
            String msg = "Expected leaf of reduce plan to " +
                "always be POStore. Found " + op.getClass().getSimpleName();
            throw new MRCompilerException(msg, errCode, PigException.BUG);
        }
    }

APACHE HADOOP版本2.8.0的HADOOP_HOME和PIG_CLASSPATH的正确值是多少？

猪堆痕迹

1 个答案: