Question

我是hadoop mapreduce编程范例的新手，有人可以告诉我如何轻松地根据值进行排序？我尝试实现另一个比较器类，但是有一种更简单的方法，比如通过作业配置来根据reducer的值进行排序。基本上我正在阅读日志文件，我想按升序订购网址到hitcount。

public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable> {

private final static IntWritable ONE = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context
                ) throws IOException, InterruptedException {
    String[] split = value.toString().split(" ");
    for(int i=0; i<split.length; i++){
        if (i==6)
            word.set(split[i]);
            context.write(word, ONE);
    }
}
}

public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> {       
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values,  Context context) throws IOException, InterruptedException {
  int sum = 0;
  for (IntWritable val : values) {
    sum += val.get();
  }
  result.set(sum);
  context.write(key, result);      
}
}

Answer 1

在reducer类中声明一个映射，并将键和值放在map中。现在在reducer类的cleanup（）方法中尝试按值对映射进行排序，然后最后在context.write（key，value）中给出值;

public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> {       
private IntWritable result = new IntWritable();

TreeMap<Text,IntWritable>result=new TreeMap<Text, IntWritable>();

public void reduce(Text key, Iterable<IntWritable> values,  Context context) throws IOException, InterruptedException {
  int sum = 0;
  for (IntWritable val : values) {
    sum += val.get();
  }
 result.put(new Text(key),new IntWritable(sum));
}
}

    @Override
    protected void cleanup(Context context)
            throws IOException, InterruptedException {

        Set<Entry<Text, IntWritable>> set = result.entrySet();
        List<Entry<Text, IntWritable>> list = new ArrayList<Entry<Text,IntWritable>>(set);
        Collections.sort( list, new Comparator<Map.Entry<Text, IntWritable>>()
        {
            public int compare( Map.Entry<Text, IntWritable> o1, Map.Entry<Text,IntWritable> o2 )
            {
                return (o2.getValue()).compareTo( o1.getValue() );
            }
        });
        for(Map.Entry<Text,IntWritable> entry:list){

            context.write(entry.getKey(),entry.getValue());
        }

    }
    }

Answer 2

在这种情况下，您必须编写两个map-reduce作业。第一份工作需要统计网址。像fisrt工作的输出将是 -

yahoo.com,100
google.com,200 
msn.com,50

将此传递给第二个map reduce工作并根据计数对其进行排序。

基于reducer值的升序排序

2 个答案: