Question

我正在尝试使用MRUnit 1.0.0来测试Hadoop v2 Reducer，但在尝试时遇到异常：

java.lang.IncompatibleClassChangeError: 
    Found class org.apache.hadoop.mapreduce.TaskInputOutputContext, but interface was expected
                at org.apache.hadoop.mrunit.internal.mapreduce.AbstractMockContextWrapper.createCommon(AbstractMockContextWrapper.java:59)
                at org.apache.hadoop.mrunit.internal.mapreduce.MockReduceContextWrapper.create(MockReduceContextWrapper.java:76)
                at org.apache.hadoop.mrunit.internal.mapreduce.MockReduceContextWrapper.<init>(MockReduceContextWrapper.java:67)
                at org.apache.hadoop.mrunit.mapreduce.ReduceDriver.getContextWrapper(ReduceDriver.java:159)
                at org.apache.hadoop.mrunit.mapreduce.ReduceDriver.run(ReduceDriver.java:142)
                at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:574)
                at org.apache.hadoop.mrunit.TestDriver.runTest(TestDriver.java:561)

我认为这意味着我在某种程度上不匹配Hadoop API的版本as in this SO question，但我不确定问题出在哪里。我正在使用Maven来引入依赖性，使用repo.hortonworks.com中的Hadoop 2.2.0.2.0.6.0-76和repo1.maven.org中的MRUnit 1.0.0：

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>2.2.0.2.0.6.0-76</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-hdfs</artifactId>
    <version>2.2.0.2.0.6.0-76</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-core</artifactId>
    <version>2.2.0.2.0.6.0-76</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-common</artifactId>
    <version>2.2.0.2.0.6.0-76</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
    <version>2.2.0.2.0.6.0-76</version>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-yarn-common</artifactId>
    <version>2.2.0.2.0.6.0-76</version>
</dependency>
<dependency>
    <groupId>org.apache.mrunit</groupId>
    <artifactId>mrunit</artifactId>
    <version>1.0.0</version>
    <classifier>hadoop2</classifier>
</dependency>

测试用例如下：

@Test
public void testReducer() throws IOException, InterruptedException {
    HH.Reduce r = new HH.Reduce();

    T1 fx1 = new T1();
    T1 fx2 = new T1();

    List<T1> values = new ArrayList<T1>();
    values.add(fx1);
    values.add(fx2);

    T1 fxBoth = new T1(fx1.size() + fx2.size());
    fxBoth.addValues(fx1);
    fxBoth.addValues(fx2);


    ReduceDriver<NullWritable, T1, NullWritable, T1> reduceDriver = ReduceDriver.newReduceDriver(r);

    reduceDriver.withInput(NullWritable.get(), values);
    reduceDriver.withOutput(NullWritable.get(), fxBoth);

    // TODO I can't seem to get this test to work.  
    // Not sure what I'm doing wrong, whether it's a real 
    // problem or a testing problem.
    reduceDriver.runTest();
}

在其他地方，在HH包中，Reduce被定义为一个非常简单的内部类：

public static class Reduce extends Reducer<NullWritable, T1, NullWritable, T1> {
    @Override
    public void reduce(NullWritable key, Iterable<T1> values, Context context)
        throws InterruptedException, IOException {

        // Need to create a new record here, because the one we're handed
        // may be recycled by our overlords.
        T1 out = new T1();
        for (T1  t : values) {
            out.addValues(t);
        }
        context.write(key, out);
    }
}

看到任何不可思议的东西？ MRUnit是否尝试使用较旧/较新版本的API？

Answer 1

我相信我有同样的问题，但我使用hadoop-core.1.2.1和mrunit-hadoop2-1.1.0。检查maven依赖项中的版本和分类器（用于测试，而不是在pom.xml中声明的那些）。

Answer 2

mrunit maven依赖中的分类器部分非常重要。

如你所说，你正在使用hadoop-core.1.2.1，TaskAttemptContext是该jar中的一个类。所以你需要在mrunit的maven依赖中将分类器设置为hadoop1。然后这没有任何问题。

如果将分类器设置为hadoop2，则需要最新的api，其中TaskAttemptContext是接口。您只需在junit中运行该文件并检查结果即可。

找到类org.apache.hadoop.mapreduce.TaskInputOutputContext，但接口是预期的

2 个答案: