加载数据时猪的错误

Question

我正在使用ubuntu 12.02 32bit并成功安装了 hadoop2.2.0 和 pig 0.12 。 Hadoop在我的系统上正常运行。

但是，每当我运行此命令时：

data = load 'atoz.csv' using PigStorage(',')  as (aa1:int, bb1:int, cc1:int, dd1:chararray);            
dump data;

我收到以下错误：

ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error whiletrying to run jobs.java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected.

这是完整的堆栈跟踪：

> 2014-01-23 10:41:44,998 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher- 1 map-reduce job(s) waiting for submission.
>             2014-01-23 10:41:45,000 [Thread-9] INFO  org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM
> Metrics with processName=JobTracker, sessionId= - already initialized
>             2014-01-23 10:41:45,001 [Thread-9] ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while
> trying to run jobs.
>             java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
>             at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>             at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>             at java.security.AccessController.doPrivileged(Native Method)
>             at javax.security.auth.Subject.doAs(Subject.java:415)
>             at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>             at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
>             at java.lang.Thread.run(Thread.java:724)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
>             2014-01-23 10:41:45,498 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
>             2014-01-23 10:41:45,502 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - job null has failed! Stop running all dependent jobs
>             2014-01-23 10:41:45,503 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
>             2014-01-23 10:41:45,507 [main] ERROR org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to
> recreate exception from backend error: Unexpected System Error
> Occured: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
>             at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>             at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>             at java.security.AccessController.doPrivileged(Native Method)
>             at javax.security.auth.Subject.doAs(Subject.java:415)
>             at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>             at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
>             at java.lang.Thread.run(Thread.java:724)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
2014-01-23 10:41:45,507 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s)
> failed!
>             2014-01-23 10:41:45,507 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode.
> Stats reported below may be incomplete
>             2014-01-23 10:41:45,508 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion    PigVersion    UserId    StartedAt    FinishedAt    Features
>             2.2.0    0.10.1    hardik    2014-01-23 10:41:44    2014-01-23 10:41:45    UNKNOWN
 Failed!
Failed Jobs:
JobId    Alias    Feature    Message    Outputs
N/A    aatoz    MAP_ONLY    Message: Unexpected System Error Occured: java.lang.IncompatibleClassChangeError: Found interface
> org.apache.hadoop.mapreduce.JobContext, but class was expected
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:225)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.checkOutputSpecs(PigOutputFormat.java:186)
>             at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:456)
>             at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:342)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>             at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>             at java.security.AccessController.doPrivileged(Native Method)
>             at javax.security.auth.Subject.doAs(Subject.java:415)
>             at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
>             at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
>             at org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:240)
>             at java.lang.Thread.run(Thread.java:724)
>             at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:260)
>             file:/tmp/temp1979716161/tmp-189979005,
Input(s):
Failed to read data from "file:///home/hardik/pig10/bin/input/atoz.csv"
Output(s):
             Failed to produce result in "file:/tmp/temp1979716161/tmp-189979005"
Job DAG:
null
2014-01-23 10:41:45,509 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
 Failed! 2014-01-23 10:41:45,510 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator
> for alias aatoz
>             Details at logfile: /home/hardik/pig10/bin/pig_1390453192689.log
>     </i>

Answer 1

Apache Pig 0.12.0默认使用旧版本的Hadoop。你必须重新编译Pig for Hadoop 2.2.0并用新的猪-0.12.1-SNAPSHOT.jar和pig-0.12.1-SNAPSHOT-withouthadoop.jar替换两个罐子。

要重新编译解压缩存档，请转到“pig-0.12.0”目录并运行：

ant clean jar-all -Dhadoopversion=23

Answer 2

如果你在项目中使用maven依赖项，仅使用命令“ant clean jar-all -Dhadoopversion = 23”构建是不够的。您需要在本地maven repo中安装由此创建的jar，或者在pom.xml中使用此依赖项（注意hadoop2的“classifier”标记）

<dependency>
<groupId>org.apache.pig</groupId>
<artifactId>pig</artifactId>
<classifier>h2</classifier>
<version>0.13.0</version>
</dependency>

Answer 3

我以另一种方式解决了这个问题。

我在CDH4.4上遇到了同样的问题。当我的Pig脚本从我使用Maven编译的java项目调用UDF时，Pig 0.11.0。 我访问过/usr/lib/pig/conf/build.properties文件。验证了针对hadoop-core，hadoop-common，hadoop-mapreduce属性提到的版本。确保所有具有相同版本的工件都作为依赖项包含在我的java项目的POM.xml文件中。

（Infact hadoop-mapreduce根据http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/4.2.0/CDH4-Installation-Guide/cdh4ig_topic_31.html有6个工件ID。我在依赖列表中包含了所有这些ID。

使用这些POM设置构建项目的jar文件后，Pig脚本能够毫无问题地调用UDF。

Answer 4

这是你的线索ERROR org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl - Error while trying to run jobs. java.lang.IncompatibleClassChangeError: Found interface org.apache.hadoop.mapreduce.JobContext, but class was expected。 接口“jobcontrol”正在查找某个类，但找不到它。

Answer 5

刚刚发现同样的异常，试图在Hadoop 2x上使用我的UDF运行脚本。

在最新版本0.14和当前主干0.15中，缺少ant目标 jar-all 。因此，如果您从源代码构建，则可以使用

  

jar-h12为Hadoop 1和Hadoop 2创建猪

或

ant jar -Dhadoopversion=23

或只是从Maven Central下载兼容版本。

在我的情况下，它是猪-0.14.0-h2.jar

Answer 6

任何高于2.2的hadoop版本也会出现此问题。更简单的方法是安装Pig版本0.15。他们在这个版本中解决了这个问题。