I am new to BigData I have written a MR program and trying to write test cases for it using MRUnit from https://dzone.com/articles/testing-mapreduce-mrunit
但我的MR程序有2个映射器和1个reducer,所以我无法使用
创建驱动程序对象newMapReduceDriver()
newMapReduceDriver(mapper,reducer)
newMapReduceDriver(mapper,reducer,combiner)
或
newMultipleInputMapReduceDriver()
newMultipleInputMapReduceDriver(combiner,reducer)
newMultipleInputMapReduceDriver(reducer)
请建议我做错事的任何其他方式。谢谢提前
这是代码
public class UC_RJoinerTool extends Configured implements Tool{
public int run(String[] args) throws Exception {
if(args == null || args.length < 4 ){
System.err.println("Usage: <User file Input Path> <Comments file Input Path> <Output Path> <Inner/Right Outer/Full join>");
ToolRunner.printGenericCommandUsage(System.err);
return -1;
} else {
Job job = Job.getInstance(getConf(), "Mapping Users with Comments");
job.setJarByClass(UC_RJoinerTool.class);
Path userInputPath = new Path(args[0]);
Path commentsInputPath = new Path(args[1]);
Path outPutPath = new Path(args[2]);
String joinTypeInput = args[3];
MultipleInputs.addInputPath(job, userInputPath, TextInputFormat.class,UserDotXmlMapper.class);
MultipleInputs.addInputPath(job, commentsInputPath, TextInputFormat.class,CommentsDotXmlMapper.class);
FileOutputFormat.setOutputPath(job,new Path(args[1]));
//When you are using TextInputFormat explicitly say the map key and value types
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.getConfiguration().set("joinType",joinTypeInput);
job.setReducerClass(UserCommentsReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
FileOutputFormat.setOutputPath(job, outPutPath);
return job.waitForCompletion(true)?0:1;
}
}
public static void main(String[] args)throws Exception {
int exitCode = ToolRunner.run(new UC_RJoinerTool(),args);
System.exit(exitCode);
}
}
我的单元testCase代码
public class UCJoinTest {
private MapDriver<LongWritable,Text,Text,Text> mapUsersDriver,mapCommentsDriver;
private ReduceDriver<Text, Text, Text,Text> reduceUCDriver;
//private MapReduceDriver<LongWritable,Text,Text, Text, Text,Text> mapReduceUCDriver;
private MultipleInputsMapReduceDriver<LongWritable,Text,Text,Text> mapReduceUCDriver;
private MapDriver<LongWritable, Text, LongWritable,Text> mapSortDriver;
private ReduceDriver<LongWritable,Text,Text,Text> reduceSortDriver;
private MapReduceDriver<LongWritable,Text,LongWritable,Text,Text,Text> mapReduceSortDriver;
@Before
public void setUp() throws Exception {
final UserDotXmlMapper usersMapper = new UserDotXmlMapper();
final CommentsDotXmlMapper CommentsMapper = new CommentsDotXmlMapper();
final UserCommentsReducer ucReducer = new UserCommentsReducer();
final ReputationSorterMapper sortMapper = new ReputationSorterMapper();
final ReputationSorterReducer sortReducer = new ReputationSorterReducer();
mapUsersDriver = MapDriver.newMapDriver(usersMapper);
mapCommentsDriver = MapDriver.newMapDriver(CommentsMapper);
reduceUCDriver = ReduceDriver.newReduceDriver(ucReducer);
mapReduceUCDriver = MapReduceDriver.newMapReduceDriver(usersMapper,CommentsMapper,ucReducer);
mapReduceUCDriver = MultipleInputsMapReduceDriver.newMultipleInputMapReduceDriver(usersMapper,CommentsMapper,ucReducer);
mapSortDriver = MapDriver.newMapDriver(sortMapper);
reduceSortDriver = ReduceDriver.newReduceDriver(sortReducer);
mapReduceSortDriver = MapReduceDriver.newMapReduceDriver(sortMapper,sortReducer);
}
public class CommentsDotXmlMapper extends Mapper<LongWritable,Text,Text,Text>{
}
public class UserDotXmlMapper extends Mapper<LongWritable,Text,Text,Text>{
}
public class UserCommentsReducer extends Reducer<Text, Text, Text,Text>{
}
出于某种原因,Stackowerflow不允许我发帖提问,所以我添加此评论请忽略此