减速器不工作或从不打电话

时间:2016-05-26 12:01:55

标签: java eclipse hadoop reduce mapper

驱动程序类:

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class DRIVER {


public static void main(String arg[]) 
{
    try{
    Path in = new Path("aamazon.txt");
    Path out = new Path("/output");

    Configuration conf = new Configuration();

    Job job = Job.getInstance(conf);
    job.setJarByClass(DRIVER.class);
    job.setMapperClass(MAPPER.class);
    job.setReducerClass(REDUCER.class);
    job.setNumReduceTasks(0);

    FileInputFormat.addInputPath(job, in);
    FileOutputFormat.setOutputPath(job, out);

    job.waitForCompletion(true);

    System.out.println("Successful");}

    catch(Exception e){
        System.out.println(e.getMessage());
    }


}
}

Mapper类:

import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class MAPPER extends Mapper<LongWritable,Text,LongWritable,Text>{

@Override
public void map(LongWritable key,Text value,Context con) 
{
    try
    {

        System.out.println(key +"\n"+ value);
        con.write(key, value);

    }

    catch(Exception e)
    {
        System.out.println(e.getMessage()); 
    }
}
}

减速机等级:

import java.io.IOException;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class REDUCER extends Reducer<LongWritable,Text,LongWritable,Text>{

@Override
public void reduce(LongWritable key,Iterable<Text> value , Context con) 
{
    System.out.println("reducer");
    try{
    for(Text t:value)
    {

        con.write(key, t);
    }
    }
    catch (Exception e)
    {
        System.out.println(e.getMessage());
    }
}
}

问题:

  1. 执行直到Mapper
  2. 永远不会被称为减速器
  3. 如果我设置setNumReduceTasks(0),那么Mapper就不会被调用
  4. 任何想法有什么不对?

1 个答案:

答案 0 :(得分:0)

您已将减少任务的数量设置为零。

    Job job = Job.getInstance(conf);
    job.setJarByClass(DRIVER.class);
    job.setMapperClass(MAPPER.class);
    job.setReducerClass(REDUCER.class);
    job.setNumReduceTasks(0); // this should be greater than 0

即便如此,如果它不起作用,请检查您是否拥有&#34; / output&#34;以下行的路径 -

    Path out = new Path("/output"); // it is in the root folder. change it to "./output"