嗨我发现地图缩减chaining有点问题。我必须形成像这样的链
mapper-> reducer->映射器
从我的第一个mapper到reducer,流程一直很好,这个reducer的输出数据不会正确地映射到下一个map.t这是一个简单的代码示例,我试过了
这是我的第一张映射器
public void map(LongWritable key, Text value,
OutputCollector<Text, IntWritable> outputCollector, Reporter reporter)
throws IOException {
String maxSalary = value.toString().split(",")[4];
outputCollector.collect(new Text("max salary"),new IntWritable(Integer.parseInt(maxSalary)));
}
这是我的减速机
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> outputCollector, Reporter reporter)
throws IOException {
int maxSalary = Integer.MIN_VALUE;
while(values.hasNext()){
maxSalary = Math.max(maxSalary, values.next().get());
}
outputCollector.collect(key, new IntWritable(maxSalary));
}
这是我的下一个简单的Mapper
public void map(Text key, IntWritable value,
OutputCollector<Text, IntWritable> outputCollector, Reporter reporter)
throws IOException {
System.out.println(value.toString());
}
这是我运行作业的主要课程
JobConf jobConf = new JobConf(jobrunner.class);
jobConf.setJobName("Chaining");
FileInputFormat.setInputPaths(jobConf, new Path("hdfs://localhost:9000/employee_data.txt"));
FileOutputFormat.setOutputPath(jobConf,new Path("hdfs://localhost:9000/chain9.txt"));
JobConf conf1 = new JobConf(false);
ChainMapper.addMapper(jobConf,chainmap.class,LongWritable.class,Text.class,Text.class,IntWritable.class,true,conf1);
JobConf conf2 = new JobConf(false);
ChainReducer.setReducer(jobConf, chainreduce.class,Text.class,IntWritable.class,Text.class,IntWritable.class,true,conf2);
JobConf conf3 = new JobConf(false);
ChainMapper.addMapper(jobConf, nextchainmap.class, Text.class,IntWritable.class,Text.class,IntWritable.class,true,conf3);
JobClient.runJob(jobConf);
我将在我的减速机中获得最高员工薪水,这必须传递到下一个映射器,在那里它会找到具有最大工资值的员工记录,我怎样才能在下一个映射器中实现这一点?任何想法?
答案 0 :(得分:2)
要连接第二张映射器,您需要拨打ChainReducer.addMapper(...)
而不是ChainMapper.addMapper(...)
。