我正在尝试使用MRunit来测试我的sortComparatorClass
。似乎MRunit应该能够使用setKeyOrderComparator
方法执行此操作,但是当我运行mapReduceDriver
时,它不会调用compare()
类的SortComparator
方法。
很确定我在使用MRunit API做错了。
这是我的单元测试代码:
public class UnitTests {
private static transient Log log = LogFactory.getLog(UnitTests.class);
MapReduceDriver<Text, Text, Text, Text, Text, Text> mapReduceDriver;
MapDriver<Text, Text, Text, Text> mapDriver;
ReduceDriver<Text, Text, Text, Text> reduceDriver;
@Before
public void setUp() throws InterruptedException, IOException {
mapDriver = new MapDriver<Text, Text, Text, Text>();
mapDriver.setMapper(new TestMapper());
reduceDriver = new ReduceDriver<Text, Text, Text, Text>();
reduceDriver.setReducer(new TestReducer());
mapReduceDriver = new MapReduceDriver(new TestMapper(), new TestReducer());
mapReduceDriver.setKeyOrderComparator(new TestSortCompartor());
}
@Test
public void testSort() throws IOException {
Text inputKey1 = new Text("def");
Text inputKey2 = new Text("abc");
Text inputValue = new Text("BlahBlahBlah");
mapReduceDriver.addInput(inputKey1, inputValue);
mapReduceDriver.addInput(inputKey2, inputValue);
List<Pair<Text, Text>> output = mapReduceDriver.run();
log.info("Got output of size "+output.size()+" with first pair = "+output.get(0).toString());
}
}
这是我的测试sortComparator
:
public class TestSortCompartor extends WritableComparator{
private static transient Log log = LogFactory.getLog(TestSortCompartor.class);
public TestSortCompartor() {
super(Text.class, true);
}
@SuppressWarnings("rawtypes")
@Override
public int compare(WritableComparable w1, WritableComparable w2) {
log.info("calling compare with key1 = "+w1.toString()+" and key2 "+w2.toString());
return w1.compareTo(w2) * -1;
}
}
当我运行测试时,我得到了这个输出:
INFO 2014-01-13 09:34:27,362 [main] (com.gradientx.gxetl.testMR.UnitTests:53) -
Got output of size 2 with first pair = (abc, BlahBlahBlah)
但是sortComparator
类没有输出 - 而且它没有反向排序键,所以我知道我的comparator()
方法没有被调用。
任何人都可以建议我做错了什么?是否可以使用MRunit来测试我自己的比较器类?是否有更好的方法为自定义比较器类进行单元测试?
仅供参考,以下是我Pom的相关依赖:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>2.0.0-mr1-cdh4.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>2.0.0-mr1-cdh4.4.0</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>1.0.0</version>
<classifier>hadoop2</classifier>
</dependency>