我希望我的第一个reduce任务产生smth(当然,< sum,count>); 在第二次减少任务中,我将计算每门课程的总和/计数。 第一个减速器任务充当合并器,求和计数;第二减少任务找到每门课程的平均值和输出平均值。我只是找不到将输出值存储为密钥对的最佳类型,然后能够对它们进行检索和计算。 HashMap不起作用。
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
import java.util.StringTokenizer;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.FloatWritable;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.Mapper.Context;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;
public class AvgGrading {
public static void main(String[] args) throws IllegalArgumentException, IOException, ClassNotFoundException, InterruptedException {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "avg grading");
job.setJarByClass(AvgGrading.class);
job.setMapperClass(MapForAverage.class);
job.setCombinerClass(ReduceForAverage.class);
job.setNumReduceTasks(2);
job.setReducerClass(ReduceForFinal.class);
job.setMapOutputKeyClass(LongWritable.class);
job.setMapOutputValueClass(Object.class);
job.setOutputKeyClass(LongWritable.class);
job.setOutputValueClass(FloatWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
public static class MapForAverage extends Mapper<LongWritable, Text, LongWritable, Object> {
public void map(LongWritable key, Text value, Context con) throws IOException, InterruptedException {
String [] word = value.toString().split(", ");
float grade = Integer.parseInt(word[1]);
int course = Integer.parseInt(word[0]);
Map <Float,Long> m = new HashMap<Float,Long>();
m.put(grade, (long) 1);
con.write(new LongWritable(course), m);
}
}
public static class ReduceForAverage extends Reducer<LongWritable, Object, LongWritable, Object> {
private FloatWritable result = new FloatWritable();
public void reduce(LongWritable course, Map<Float,Long> values, Context con)
throws IOException, InterruptedException {
Map <Float,Long> m = new HashMap<Float,Long>();
float sum = 0;
long count =0;
for (Map.Entry<Float, Long> entry : values.entrySet()) {
sum += entry.getKey();
count++;
}
m.put(sum, count);
con.write(course, m);
}
}
public static class ReduceForFinal extends Reducer<LongWritable, Object, LongWritable, FloatWritable> {
private FloatWritable result = new FloatWritable();
public void reduce(LongWritable course, Map<Long,Float>values, Context con)
throws IOException, InterruptedException {
long key = 0;
float value=0;
for ( Map.Entry<Long, Float> entry : values.entrySet()) {
key = entry.getKey();
value = entry.getValue();
}
float res= key/value;
con.write(course, new FloatWritable(res));
}
}
}
请注意,我无法在Reduce任务中迭代Iterable < Map<Float,Int>>
,所以我传递的是简单的Map,这可能不正确。
错误代码是:
Unable to initialize MapOutputCollector org.apache.hadoop.mapred.MapTask$MapOutputBuffer
显示java.lang.NullPointerException
2nd Reducer失败
答案 0 :(得分:0)
Map没有实现Writable,你说你的组合器和reducer输入值的类是Object,而你正在发射Map。你只需要为此目的创建一个自定义类。请记住,如果要在hadoop中发出一些内容,则自定义类必须实现Writable。这是你可以做的事情:
public class Counter implements Writable {
private float sum;
private long count;
public Counter(float sum, long count){
this.sum = sum;
this.count = count;
}
/* Methods to get and set private variables of the class */
public float getSum() {
return sum;
}
public void setSum(float sumValue) {
sum=sumValue;
}
public long getCount() {
return count;
}
public void setCount(long countValue) {
count=countValue;
}
/* Methods to serialize and deserialize the contents of the
instances of this class */
@Override /* Serialize the fields of this object to out */
public void write(DataOutput out) throws IOException{
out.writeFloat(sum);
out.writeLong(count);
}
@Override /* Deserialize the fields of this object from in */
public void readFields(DataInputin) throws IOException{
sum=in.readFloat();
count=in.readLong();
}
}
所以在你的第一个映射器中,你可以用这种方式创建和发出一个计数器:
Counter counter = new Counter(grade, 1);
con.write(course, counter);
此时,在您的第一个减速器中,您将拥有一个表示该过程的键和一个可迭代的值,该值对于所有计数器都是可迭代的,并且通过此迭代,您可以计算平均值。请记住更新mapper和reducers类参数以与新的一致。