Hadoop Map-reduce编程语法错误

时间：2014-03-21 13:44:58

标签： hadoop mapreduce

我的输入是很多文本文件。我希望我的map-reduce程序将所有文件名和相关句子与文件名一起写入一个输出文件中，我想从映射器中发出文件名（键）和相关句子（值） 。 reducer将收集键和所有值，并在输出中写入文件名及其相关句子。

这是我的mapper和reducer的代码：

import java.io.IOException;

   import java.util.*;
import org.apache.hadoop.fs.Path;
   import org.apache.hadoop.io.*;
    import org.apache.hadoop.mapred.*;


      public class WordCount {

public static class Map extends MapReduceBase implements Mapper<LongWritable,   
    Text, Text, Text> {



  public void map(LongWritable key, Text value, OutputCollector<Text,Text> 
      output, Reporter reporter) throws IOException {
  String filename = new String();
  FileSplit filesplit = (FileSplit)reporter.getInputSplit();
  filename=filesplit.getPath().getName();
      output.collect(new Text(filename), value);


     }

    }


    public static class Reduce extends MapReduceBase implements Reducer<Text, Text,  
    Text, Text> {

     public void reduce(Text key, Iterable<Text> values, OutputCollector<Text, 
         Text> output, Reporter reporter) throws IOException {

    StringBuilder builder = new StringBuilder();
    for(Text value : values)
    {
        String str = value.toString();
        builder.append(str);
    }
    String valueToWrite=builder.toString();
    output.collect(key, new Text(valueToWrite));
      }

    @Override
    public void reduce(Text arg0, Iterator<Text> arg1,
            OutputCollector<Text, Text> arg2, Reporter arg3)
            throws IOException {
                    }




  }
   public static void main(String[] args) throws Exception {
  JobConf conf = new JobConf(WordCount.class);
  conf.setJobName("wordcount");

  conf.setMapperClass(Map.class);
  conf.setReducerClass(Reduce.class);
  conf.setJarByClass(WordCount.class);
  conf.setOutputKeyClass(Text.class);
  conf.setOutputValueClass(Text.class);

  conf.setInputFormat(TextInputFormat.class);
  conf.setOutputFormat(TextOutputFormat.class);
  conf.setNumReduceTasks(1);
  FileInputFormat.setInputPaths(conf, new Path(args[0]));
  FileOutputFormat.setOutputPath(conf, new Path(args[1]));

  JobClient.runJob(conf);

 }
 }

输出终端如下

 14/03/21 00:38:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library   
  for your platform... using builtin-java classes where applicable
  14/03/21 00:38:27 WARN mapred.JobClient: Use GenericOptionsParser for parsing the 
 arguments. Applications should implement Tool for the same.
14/03/21 00:38:27 WARN mapred.JobClient: No job jar file set.  User classes may not  
be found. See JobConf(Class) or JobConf#setJar(String).
14/03/21 00:38:27 WARN snappy.LoadSnappy: Snappy native library not loaded
14/03/21 00:38:27 INFO mapred.FileInputFormat: Total input paths to process : 2
14/03/21 00:38:27 INFO mapred.JobClient: Running job: job_local_0001
14/03/21 00:38:27 INFO util.ProcessTree: setsid exited with exit code 0
14/03/21 00:38:27 INFO mapred.Task:  Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@4911b910
14/03/21 00:38:27 INFO mapred.MapTask: numReduceTasks: 1
14/03/21 00:38:27 INFO mapred.MapTask: io.sort.mb = 100
14/03/21 00:38:27 INFO mapred.MapTask: data buffer = 79691776/99614720
14/03/21 00:38:27 INFO mapred.MapTask: record buffer = 262144/327680
14/03/21 00:38:27 INFO mapred.MapTask: Starting flush of map output
14/03/21 00:38:27 INFO mapred.MapTask: Finished spill 0
14/03/21 00:38:27 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And  
is in the process of commiting
14/03/21 00:38:28 INFO mapred.JobClient:  map 0% reduce 0%
14/03/21 00:38:30 INFO mapred.LocalJobRunner:  
file:/root/Desktop/wordcount/sample.txt:0+5371
14/03/21 00:38:30 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
14/03/21 00:38:30 INFO mapred.Task:  Using ResourceCalculatorPlugin :  
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1f8166e5
14/03/21 00:38:30 INFO mapred.MapTask: numReduceTasks: 1
14/03/21 00:38:30 INFO mapred.MapTask: io.sort.mb = 100
14/03/21 00:38:30 INFO mapred.MapTask: data buffer = 79691776/99614720
14/03/21 00:38:30 INFO mapred.MapTask: record buffer = 262144/327680
14/03/21 00:38:30 INFO mapred.MapTask: Starting flush of map output
14/03/21 00:38:30 INFO mapred.MapTask: Finished spill 0
14/03/21 00:38:30 INFO mapred.Task: Task:attempt_local_0001_m_000001_0 is done. And      
is in the process of commiting
14/03/21 00:38:31 INFO mapred.JobClient:  map 100% reduce 0%
14/03/21 00:38:33 INFO mapred.LocalJobRunner:  
file:/root/Desktop/wordcount/sample.txt~:0+587
14/03/21 00:38:33 INFO mapred.Task: Task 'attempt_local_0001_m_000001_0' done.
14/03/21 00:38:33 INFO mapred.Task:  Using ResourceCalculatorPlugin : 
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3963b3e
14/03/21 00:38:33 INFO mapred.LocalJobRunner: 
14/03/21 00:38:33 INFO mapred.Merger: Merging 2 sorted segments
14/03/21 00:38:33 INFO mapred.Merger: Down to the last merge-pass, with 2 segments  
left of total size: 7549 bytes
14/03/21 00:38:33 INFO mapred.LocalJobRunner: 
14/03/21 00:38:33 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And  
is in the process of commiting
14/03/21 00:38:33 INFO mapred.LocalJobRunner: 
14/03/21 00:38:33 INFO mapred.Task: Task attempt_local_0001_r_000000_0 is allowed to 
commit now
14/03/21 00:38:33 INFO mapred.FileOutputCommitter: Saved output of task  
'attempt_local_0001_r_000000_0' to file:/root/Desktop/wordcount/output
14/03/21 00:38:36 INFO mapred.LocalJobRunner: reduce > reduce
14/03/21 00:38:36 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
14/03/21 00:38:37 INFO mapred.JobClient:  map 100% reduce 100%
14/03/21 00:38:37 INFO mapred.JobClient: Job complete: job_local_0001
14/03/21 00:38:37 INFO mapred.JobClient: Counters: 21
14/03/21 00:38:37 INFO mapred.JobClient:   File Input Format Counters 
14/03/21 00:38:37 INFO mapred.JobClient:     Bytes Read=5958
14/03/21 00:38:37 INFO mapred.JobClient:   File Output Format Counters 
14/03/21 00:38:37 INFO mapred.JobClient:     Bytes Written=8
14/03/21 00:38:37 INFO mapred.JobClient:   FileSystemCounters
14/03/21 00:38:37 INFO mapred.JobClient:     FILE_BYTES_READ=26020
14/03/21 00:38:37 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=117337
14/03/21 00:38:37 INFO mapred.JobClient:   Map-Reduce Framework
14/03/21 00:38:37 INFO mapred.JobClient:     Map output materialized bytes=7557
14/03/21 00:38:37 INFO mapred.JobClient:     Map input records=122
14/03/21 00:38:37 INFO mapred.JobClient:     Reduce shuffle bytes=0
14/03/21 00:38:37 INFO mapred.JobClient:     Spilled Records=244
14/03/21 00:38:37 INFO mapred.JobClient:     Map output bytes=7301
14/03/21 00:38:37 INFO mapred.JobClient:     Total committed heap usage  
(bytes)=954925056
14/03/21 00:38:37 INFO mapred.JobClient:     CPU time spent (ms)=0
14/03/21 00:38:37 INFO mapred.JobClient:     Map input bytes=5958
14/03/21 00:38:37 INFO mapred.JobClient:     SPLIT_RAW_BYTES=185
14/03/21 00:38:37 INFO mapred.JobClient:     Combine input records=0
14/03/21 00:38:37 INFO mapred.JobClient:     Reduce input records=0
14/03/21 00:38:37 INFO mapred.JobClient:     Reduce input groups=2
14/03/21 00:38:37 INFO mapred.JobClient:     Combine output records=0
14/03/21 00:38:37 INFO mapred.JobClient:     Physical memory (bytes) snapshot=0
14/03/21 00:38:37 INFO mapred.JobClient:     Reduce output records=0
14/03/21 00:38:37 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
14/03/21 00:38:37 INFO mapred.JobClient:     Map output records=122

当我使用相同的inputformat（keyvaluetextinputformat.class）配置运行上面的mapper和reducer时，它不会在输出中写任何东西..

我应该改变什么来实现我的目标..

1 个答案:

答案 0 :(得分：1)

当我看到计数器时，我看到reducer没有输入记录。这意味着减速器方面发生了一些事情。

查看代码，我看到你的reduce（）方法签名是：

public void reduce(Text key, Iterable<Text> values, OutputCollector<Text, 
     Text> output, Reporter reporter) throws IOException

并且它声明值是Iterable。正确的类型是Iterator（根据在线Javadoc）：

public void reduce(Text key, Iterator<Text> values, OutputCollector<Text, 
     Text> output, Reporter reporter) throws IOException

因此即使您提供了reduce方法，但签名错误的事实意味着它没有被使用。

将Iterable更改为Iterator，它应该可以正常工作

相关问题
最新问题