Question

我在hadoop中遇到了WordCount程序的问题。单词计数不正确，它对所有单词显示0，但输出中存在所有不同的单词。

这是我的示例数据，加载到hdfs

# filename: file01.txt
Hello World Bye World

和

# filename: file02.txt
Hello Hadoop Bye Hadoop

这是来源：

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapred.*;
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.io.*;


public class WordCount {
    public static class Map
            extends MapReduceBase
            implements Mapper<LongWritable, Text, Text, IntWritable> {


        private final static IntWritable one = new IntWritable();
        private Text word = new Text();

        public void map(LongWritable longWritable, Text value,
                        OutputCollector<Text, IntWritable> output,
                        Reporter reporter) throws IOException {

            String line = value.toString();
            StringTokenizer tokenizer = new StringTokenizer(line);
            while (tokenizer.hasMoreTokens()) {
                word.set(tokenizer.nextToken());
                output.collect(word, one);
            }
        }
    }


    public static class Reduce
            extends MapReduceBase
            implements Reducer<Text, IntWritable, Text, IntWritable> {

        public void reduce(Text key, Iterator<IntWritable> values,
                           OutputCollector<Text, IntWritable> output,
                           Reporter reporter) throws IOException {

            int sum = 0;
            while(values.hasNext()) {
                sum += values.next().get();
            }
            output.collect(key, new IntWritable(sum));
        }
    }

    public static void main(String[] args) throws IOException {

        JobConf jobConf = new JobConf(WordCount.class);
        jobConf.setJobName("wordcount");

        jobConf.setOutputKeyClass(Text.class);
        jobConf.setOutputValueClass(IntWritable.class);

        jobConf.setCombinerClass(WordCount.Reduce.class);
        jobConf.setReducerClass(WordCount.Reduce.class);
        jobConf.setMapperClass(WordCount.Map.class);

        jobConf.setInputFormat(TextInputFormat.class);
        jobConf.setOutputFormat(TextOutputFormat.class);

        FileInputFormat.setInputPaths(jobConf, new Path(args[0]));
        FileOutputFormat.setOutputPath(jobConf, new Path(args[1]));

        JobClient.runJob(jobConf);
    }
}

当我运行jar时，输出文件在输出文件夹中生成，但它显示以下内容：

$ bin/hdfs dfs -cat ./output/part-00000
17/11/09 02:50:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Bye 0
Hadoop  0
Hello   0
World   0

如您所见，所有计数均为零，但我无法在实施中找到错误的位置。

Answer 1

是的，我已经尝试调试代码，错误发生在Map类

中

 public static class Map
        extends MapReduceBase
        implements Mapper<LongWritable, Text, Text, IntWritable> {


    private final static IntWritable one = new IntWritable();
    private Text word = new Text();

    public void map(LongWritable longWritable, Text value,
                    OutputCollector<Text, IntWritable> output,
                    Reporter reporter) throws IOException {

        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
            word.set(tokenizer.nextToken());
            output.collect(word, one);
        }
    }
}

由于您的Mapper类返回null（0）为Value，因此reducer无法减小值

因此初始化值1，以便为每个单词返回值。

这是代码

public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {

    private final static IntWritable one = new IntWritable();
    private Text word = new Text();

    public void map(LongWritable longWritable, Text value, OutputCollector<Text, IntWritable> output,
            Reporter reporter) throws IOException {

        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
            word.set(tokenizer.nextToken());
            one.set(1);

            output.collect(word, one);
        }
    }

它会起作用....

Hadoop WordCount为所有单词提供0计数

1 个答案: