Question

我正在尝试使用MRunit来测试我的sortComparatorClass。似乎MRunit应该能够使用setKeyOrderComparator方法执行此操作，但是当我运行mapReduceDriver时，它不会调用compare()类的SortComparator方法。
很确定我在使用MRunit API做错了。

这是我的单元测试代码：

public class UnitTests {

private static transient Log log = LogFactory.getLog(UnitTests.class);

MapReduceDriver<Text, Text, Text, Text, Text, Text> mapReduceDriver;
MapDriver<Text, Text, Text, Text> mapDriver;
ReduceDriver<Text, Text, Text, Text> reduceDriver;

@Before
public void setUp() throws InterruptedException, IOException {
    mapDriver = new MapDriver<Text, Text, Text, Text>();
    mapDriver.setMapper(new TestMapper());
    reduceDriver = new ReduceDriver<Text, Text, Text, Text>();
    reduceDriver.setReducer(new TestReducer());
    mapReduceDriver = new MapReduceDriver(new TestMapper(), new TestReducer());
    mapReduceDriver.setKeyOrderComparator(new TestSortCompartor());
}

@Test
public void testSort() throws IOException {

    Text inputKey1 = new Text("def");
    Text inputKey2 = new Text("abc");
    Text inputValue = new Text("BlahBlahBlah");
    mapReduceDriver.addInput(inputKey1, inputValue);
    mapReduceDriver.addInput(inputKey2, inputValue);

    List<Pair<Text, Text>> output = mapReduceDriver.run();
    log.info("Got output of size "+output.size()+" with first pair = "+output.get(0).toString());
}
}

这是我的测试sortComparator：

public class TestSortCompartor extends WritableComparator{
private static transient Log log = LogFactory.getLog(TestSortCompartor.class);

public TestSortCompartor() {
    super(Text.class, true);
}

@SuppressWarnings("rawtypes")
@Override
public int compare(WritableComparable w1, WritableComparable w2) {
    log.info("calling compare with key1 = "+w1.toString()+" and key2 "+w2.toString());
    return w1.compareTo(w2) * -1;
}
}

当我运行测试时，我得到了这个输出：

INFO 2014-01-13 09:34:27,362 [main] (com.gradientx.gxetl.testMR.UnitTests:53) - 
Got output of size 2 with first pair = (abc, BlahBlahBlah)

但是sortComparator类没有输出 - 而且它没有反向排序键，所以我知道我的comparator()方法没有被调用。

任何人都可以建议我做错了什么？是否可以使用MRunit来测试我自己的比较器类？是否有更好的方法为自定义比较器类进行单元测试？

仅供参考，以下是我Pom的相关依赖：

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>2.0.0-mr1-cdh4.4.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.0.0-mr1-cdh4.4.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.mrunit</groupId>
    <artifactId>mrunit</artifactId>
    <version>1.0.0</version>
    <classifier>hadoop2</classifier>
</dependency>

MRunit：如何测试自定义排序比较器？

0 个答案: