Driver中的job.setOutputKeyClass和setOutputValueClass与reducer的context.write方法不匹配,仍然程序正在运行fine.how?

时间:2014-02-23 12:50:47

标签: java hadoop mapreduce hadoop2

驱动程序代码:

public class WcDriver {

    public static void main(String[] args) throws IOException,
        InterruptedException, ClassNotFoundException {
        Configuration conf = new Configuration();

        Job job = new Job(conf, "WcDriver");
        job.setJarByClass(WcDriver.class);

        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(LongWritable.class);

        job.setInputFormatClass(TextInputFormat.class);
        job.setOutputFormatClass(TextOutputFormat.class);

        FileInputFormat.setInputPaths(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        job.setMapperClass(WcMapper.class);
        job.setReducerClass(WcReducer.class);

        job.waitForCompletion(true);
    }
}

减速器代码

public class WcReducer extends Reducer<Text, LongWritable, Text,String>
{   
    @Override
    public void reduce(Text key, Iterable<LongWritable> values, Context context) throws IOException, InterruptedException {
        String key1 = null;
        int total = 0;
        for (LongWritable value : values) {
            total += value.get();
            key1= key.toString();
        }
        context.write(new Text(key1), "ABC");
    }
}

在这里,在驱动程序类中我设置了job.setOutputKeyClass(Text.class)job.setOutputValueClass(LongWritable.class),但在reducer类中我正在编写一个字符串context.write(new Text(key1), "ABC");。我认为运行程序时应该出错,因为输出类型不匹配,而reducer的密钥也应该实现WritableComparable,值应该实现Writable接口。奇怪的是,这个程序运行良好。我不明白为什么没有例外。

2 个答案:

答案 0 :(得分:3)

尝试这样做:

 //  job.setOutputFormatClass(TextOutputFormat.class); 
// comment this line, and you'll sure get exception of casting.

这是因为,TextOutputFormat假定LongWritable为键,Text为值,如果你不定义outPutFormat类,它会期望获得writable的默认行为,这是默认情况下,但如果你提到它,它会隐式地将它强制转换为给定的类型。;

答案 1 :(得分:1)

试试这个

//job.setOutputValueClass(LongWritable.class); if you comment this line you get an error
this will for only define the key value pair by defaul it depent on the output format and
it will be text so this is not giving any error