Question

我有一个简单的Map / Reduce任务。 Mapper被正常调用并执行，但永远不会调用reducer。

配置：

conf.setJobName("Index Builder");
conf.setSpeculativeExecution(false);

FileInputFormat.setInputPaths(conf, new Path(args[0].toString()));
FileOutputFormat.setOutputPath(conf, new Path(args[1].toString()));

conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(NullOutputFormat.class);

conf.setMapperClass(IndexMapper.class);
conf.setReducerClass(IndexReducer.class);

conf.setMapOutputKeyClass(NullWritable.class);
conf.setMapOutputValueClass(NullWritable.class);
conf.setOutputValueClass(NullWritable.class);
conf.setOutputKeyClass(NullWritable.class);

Mapper签名：

public class IndexMapper extends MapReduceBase implements
  Mapper<LongWritable, Text, NullWritable, NullWritable> {

   @Override
   public void map(LongWritable key, Text val,
     OutputCollector<NullWritable, NullWritable> output,
     Reporter reporter) throws IOException {
        // MAP FUNCTION
   }

}

Reducer签名：

public class IndexReducer extends MapReduceBase implements
  Reducer<NullWritable, NullWritable, NullWritable, NullWritable> {

   @Override
   public void reduce(NullWritable arg0, Iterator<NullWritable> arg1,
     OutputCollector<NullWritable, NullWritable> arg2, Reporter reporter)
     throws IOException {
      // REDUCE CODE
   }
}

Answer 1

您为映射器设置输出格式为 NullOutputFormat ，因此它不会生成任何内容。因此，永远不会调用reducer，因为它需要运行一些数据。

Hadoop Mapreduce：从未调用过Reducer

1 个答案: