我正在尝试通过更改hadoop给出的wordcount示例来创建简单的地图缩减作业。
我正在尝试列出一个列表而不是一个单词的计数。 wordcount示例给出以下输出
hello 2
world 2
我正在尝试将其作为列表输出,这将构成未来工作的基础
hello 1 1
world 1 1
我认为我走在正确的轨道上,但我在编写列表时遇到了麻烦。而不是上述,我正在
Hello foo.MyArrayWritable@61250ff2
World foo.MyArrayWritable@483a0ab1
这是我的MyArrayWritable。我在write(DataOuptut arg0)
中放了一个sys,但它从不输出任何内容,所以我认为该方法可能不会被调用,我不知道为什么。
class MyArrayWritable extends ArrayWritable{
public MyArrayWritable(Class<? extends Writable> valueClass, Writable[] values) {
super(valueClass, values);
}
public MyArrayWritable(Class<? extends Writable> valueClass) {
super(valueClass);
}
@Override
public IntWritable[] get() {
return (IntWritable[]) super.get();
}
@Override
public void write(DataOutput arg0) throws IOException {
for(IntWritable i : get()){
i.write(arg0);
}
}
}
编辑 - 添加更多源代码
public class WordCount {
public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
context.write(word, one);
}
}
}
public static class Reduce extends Reducer<Text, IntWritable, Text, MyArrayWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException {
ArrayList<IntWritable> list = new ArrayList<IntWritable>();
for (IntWritable val : values) {
list.add(val);
}
context.write(key, new MyArrayWritable(IntWritable.class, list.toArray(new IntWritable[list.size()])));
}
}
public static void main(String[] args) throws Exception {
if(args == null || args.length == 0)
args = new String[]{"./wordcount/input","./wordcount/output"};
Path p = new Path(args[1]);
FileSystem fs = FileSystem.get(new Configuration());
fs.exists(p);
fs.delete(p, true);
Configuration conf = new Configuration();
Job job = new Job(conf, "wordcount");
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
job.setJarByClass(WordCount.class);
job.setInputFormatClass(TextInputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
}
}
答案 0 :(得分:22)
你的reducer中有一个'bug' - 值迭代器在整个循环中重复使用相同的IntWritable,所以你应该将添加到列表中的值包装如下:
public void reduce(Text key, Iterable<IntWritable> values, Context context)
throws IOException, InterruptedException {
ArrayList<IntWritable> list = new ArrayList<IntWritable>();
for (IntWritable val : values) {
list.add(new IntWritable(val));
}
context.write(key, new MyArrayWritable(IntWritable.class, list.toArray(new IntWritable[list.size()])));
}
这实际上并不是一个问题,因为你正在使用一个数组列表而你的映射器只输出一个值(一个),但如果你扩展这个代码,它可能会让你失望。
您还需要在作业中定义地图和减速器输出类型不同:
// map output types
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
// reducer output types
job.setOutputValueClass(Text.class);
job.setOutputValueClass(MyArrayWritable.class);
您可能希望明确定义reducers的数量(这可能是您从未看到sysout写入任务日志的原因,尤其是当您的集群管理员已将默认数字定义为0时):
job.setNumReduceTasks(1);
您使用默认的文本输出格式,它在输出键和值对上调用toString() - MyArrayWritable没有重写的toString方法,因此您应该在MyArrayWritable中放置一个:
@Override
public String toString() {
return Arrays.toString(get());
}
最后从MyArrayWritable中删除重写的write
方法 - 这不是与免费的readFields方法兼容的有效实现。你不需要覆盖这个方法但是如果你这样做(比如说你想看一个sysout来验证它被调用)那么就做这样的事情:
@Override
public void write(DataOutput arg0) throws IOException {
System.out.println("write method called");
super.write(arg0);
}