每当我从终端运行任何apache pig代码时,一切顺利,我得到了结果。所以我得出结论,我对Pig 0.15.0和Hadoop 2.7.0的安装没问题。 问题是当我从java代码中运行pigServer时:
PigServer pigServer = new PigServer(ExecType.MAPREDUCE, conf);
pigServer.setBatchOn();
pigServer.debugOff();
pigServer.setJobName(JobId);
pigServer.registerScript(scriptUrl, params);
pigServer.executeBatch();
我的maven依赖项是:
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.15.0</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.0</version>
</dependency>
我收到以下错误。
WARN org.apache.pig.backend.hadoop20.PigJobControl - falling back to default JobControl (not using hadoop 0.20 ?)
java.lang.NoSuchFieldException: runnerState
at java.lang.Class.getDeclaredField(Class.java:1948)
at org.apache.pig.backend.hadoop20.PigJobControl.<clinit>(PigJobControl.java:51)
at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:100)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:313)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:199)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:277)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1367)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1352)
at org.apache.pig.PigServer.execute(PigServer.java:1341)
at org.apache.pig.PigServer.executeBatch(PigServer.java:392)
at org.apache.pig.PigServer.executeBatch(PigServer.java:375)
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:170)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:232)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:203)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:479)
我曾经在Hadoop 1上运行上面的代码并且它正在运行,但现在却没有。
答案 0 :(得分:0)
默认情况下,pig使用Hadoop 0.20版本,因此运行pig时假设您使用的是Hadoop 0.20,因此您收到该错误
通过将HADOOP_HOME设置为指向已安装Hadoop的目录,可以使用不同版本的Hadoop运行Pig。如果你没有设置HADOOP_HOME,默认情况下Pig会运行嵌入式版本,目前是Hadoop 0.20.2 .--写在Apache pig网站https://pig.apache.org/docs/r0.9.2/start.html
在eclipse中设置HADOOP_HOME
Run Configurations-->ClassPath-->User Entries-->Advanced-->Add ClassPath Variables-->New-->Name(HADOOP_HOME)-->Path(You Hadoop directory path)
需要Maven依赖
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.7.1</version>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.4</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<version>0.15.0</version>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr-runtime</artifactId>
<version>3.4</version>
</dependency>
</dependencies>