Question

我为MapReduce文本排序编写了这样的代码：

public static class SortMapper extends Mapper<Object, Text, Text, Text> {
    private Text citizenship = new Text();

    @Override
    public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
        citizenship.set(value.toString().split(",")[11]);
        context.write(citizenship, value);
    }
}

public static class PrintReducer extends Reducer<Text, Text, NullWritable, Text> {

    @Override
    protected void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException {
        Iterator<Text> valIt = values.iterator();

        while (valIt.hasNext()) {
            Text value = valIt.next();
            context.write(NullWritable.get(), value);
        }
    }
}

public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = Job.getInstance(conf, "Football Sort");
    job.setJarByClass(FootballSort.class);
    job.setMapperClass(SortMapper.class);
    job.setCombinerClass(PrintReducer.class);
    job.setReducerClass(PrintReducer.class);
    job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(Text.class);
    job.setOutputKeyClass(NullWritable.class);
    job.setOutputValueClass(Text.class);
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
    System.exit(job.waitForCompletion(true) ? 0 : 1);
}

但总是抓住

第26,34行中的IOException 原因：类org.apache.hadoop.io.NullWritable不是类org.apache.hadoop.io.Text

Answer 1

您的mapper outputformat与您的代码不匹配，在您的main方法中设置输出TEXT

job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(Text.class);

但是在您的映射器public static class PrintReducer extends Reducer<Text, Text, NullWritable, Text>中，您可以设置NullWritable TEXT

Answer 2

@Abhinay：你不能在这种情况下使用合成器。合作器是迷你缩减器，其操作是可交换的和关联的，合并器的签名应该与Reducers匹配。如果合并器签名是“”，你将得到错误作为reducer输入键和值是--Text和IntWritable，但是合并者的输出键和值类是Text，NullWritable - Unmesha SreeVeni 2015年12月28日5:51

// job.setCombinerClass（PrintReducer.class）;或删除此字符串是修复方法

MapReduce IOException

2 个答案: