Question

我正在运行一个hadoop-2.2.0，伪分布式集群。我尝试使用以下代码来获取每个映射器和reducer所花费的时间，但我在这里得到了映射器和reducer的数量0 ..

JobConf conf = new JobConf(getConf(), WordCount.class);
    conf.setJobName("wordcount");


    // the keys are words (strings)
    conf.setOutputKeyClass(Text.class);
    // the values are counts (ints)
    conf.setOutputValueClass(IntWritable.class);

    conf.setMapperClass(MapClass.class);        
    conf.setCombinerClass(Reduce.class);
    conf.setReducerClass(Reduce.class);

    List<String> other_args = new ArrayList<String>();
    for(int i=0; i < args.length; ++i) {
      try {
        if ("-m".equals(args[i])) {
          conf.setNumMapTasks(Integer.parseInt(args[++i]));
        } else if ("-r".equals(args[i])) {
          conf.setNumReduceTasks(Integer.parseInt(args[++i]));
        } else {
          other_args.add(args[i]);
        }
      } catch (NumberFormatException except) {
        System.out.println("ERROR: Integer expected instead of " + args[i]);
        return printUsage();
      } catch (ArrayIndexOutOfBoundsException except) {
        System.out.println("ERROR: Required parameter missing from " +
                           args[i-1]);
        return printUsage();
      }
    }
    // Make sure there are exactly 2 parameters left.
    if (other_args.size() != 2) {
      System.out.println("ERROR: Wrong number of parameters: " +
                         other_args.size() + " instead of 2.");
      return printUsage();
    }
    FileInputFormat.setInputPaths(conf, other_args.get(0));
    FileOutputFormat.setOutputPath(conf, new Path(other_args.get(1)));

    JobClient jobclient = new JobClient(conf);
    RunningJob runjob = jobclient.submitJob(conf);          

    TaskReport[] maps = jobclient.getMapTaskReports(runjob.getID());
    System.out.println("Number of Mappers "+maps.length);
    for (TaskReport rpt : maps) {
      long duration = rpt.getFinishTime() - rpt.getStartTime();
      System.out.println("Mapper duration: " + duration);
    }

    TaskReport[] reduces = jobclient.getReduceTaskReports(runjob.getID());
     System.out.println("Number of Reducers "+reduces.length);
     for (TaskReport rpt : reduces) {
      long duration = rpt.getFinishTime() - rpt.getStartTime();
      System.out.println("Reducer duration: " + duration);
    }

    return 0;

我做错了吗？..

Answer 1

你快到了。唯一的问题是TaskReport的查询在提交的作业进行有意义的进展之前发生得太早。因此，要获得结果，以下代码将执行：

    ...
    RunningJob runjob = jobclient.submitJob(conf); 
    while (!runjob.isComplete()) {
        System.out.println("sleeping for 5 sec...");
        Thread.sleep(5000);
    }
    TaskReport[] maps = jobclient.getMapTaskReports(runjob.getID());
    ...

我如何获得每个映射器和缩减器执行时间

1 个答案: