在Hadoop的reducer设置中访问Counter

时间:2018-08-31 00:11:26

标签: java hadoop

我正在尝试通过运行

获取映射器中输入记录的行数
job.getCounters().findCounter(TaskCounter.MAP_INPUT_RECORDS).getValue()

实际上,它在工作完成后可以工作,我想在减速器的设置阶段做同样的事情。我已尝试根据此页面Accessing a mapper's counter from a reducer覆盖2个设置功能。但是,它们都不起作用,并且都返回空指针异常。

    @Override
    protected void setup(Reducer<Text, DoubleWritable, Text, DoubleWritable>.Context context)
            throws IOException, InterruptedException {
        Configuration conf = context.getConfiguration();
        Cluster cluster = new Cluster(conf);
        //RunningJob job = (RunningJob) cluster.getJob(context.getJobID());
        Job job = cluster.getJob(context.getJobID());
        System.out.println(job.getCounters().findCounter(TaskCounter.MAP_INPUT_RECORDS).getValue());
    }

    @Override
    protected void setup(Reducer<Text, DoubleWritable, Text, DoubleWritable>.Context context)
            throws IOException, InterruptedException {
        Configuration conf = context.getConfiguration();
        JobClient client = new JobClient(conf);
        RunningJob job = client.getJob(JobID.forName(conf.get("mapred.map.id")));
        //Job job = (Job) client.getJob((JobID) context.getJobID());
        System.out.println(job.getCounters().findCounter(TaskCounter.MAP_INPUT_RECORDS).getValue());

    }

有人可以帮我解决问题吗?非常感谢。

0 个答案:

没有答案