Question

我正在尝试通过oozie工作运行wordcount程序当我像hadoop jar wordcoutjar /data.txt /out一样手动运行wordcout jar时。它运行正常并给我输出这是我的wordcount程序的映射器代码的详细信息。

    public class MapperWordcount extends Mapper<LongWritable, Text, Text, IntWritable>{
        private final static IntWritable one = new IntWritable(1);
        private Text word = new Text();

        public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            String line = value.toString();
            StringTokenizer tokenizer = new StringTokenizer(line);
            while (tokenizer.hasMoreTokens()) {
                word.set(tokenizer.nextToken());
                context.write(word, one);
            }
        }

    }

当我通过oozie作业执行时，错误如下：

 2015-07-31 00:39:23,357 FATAL [IPC Server handler 29 on 40854] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1438294006985_0011_m_000000_3 - exited : java.lang.RuntimeException: Error in configuring object
            at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
            at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
            at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
            at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446)
            at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
            at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
            at java.security.AccessController.doPrivileged(Native Method)
            at javax.security.auth.Subject.doAs(Subject.java:415)
            at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
            at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
    Caused by: java.lang.reflect.InvocationTargetException
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
            ... 9 more
    Caused by: java.lang.RuntimeException: java.lang.RuntimeException: class com.mr.wc.MapperWordcount not org.apache.hadoop.mapred.Mapper
            at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2108)
            at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:1109)
            at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
            ... 14 more
    Caused by: java.lang.RuntimeException: **class com.mr.wc.MapperWordcount not org.apache.hadoop.mapred.Mapper**
            at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2102)
            ... 16 more

我的pom.xml是这样的。

   <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-client</artifactId>
        <version>2.6.0</version>
    </dependency>
   <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-hdfs</artifactId>
        <version>2.6.0</version>
</dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-core</artifactId>
        <version>2.6.0</version>

Answer 1

我在这里遇到了同样的问题，实际问题是代码是指旧地图减少库，而在运行时它正试图找到新的地图减少图书馆。

在Gradle

compile("org.apache.hadoop:hadoop-core:2.4.0")

并在你的pom.xml中

<dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>2.4.0</version>
    </dependency>

将Mapper和reducer中的所有引用从org.apache.hadoop.mapred.Mapper更改为org.apache.hadoop.mapreduce.Mapper

Oozie作业在通过hue运行时因“not org.apache.hadoop.mapred.Mapper”而失败

1 个答案: