为异常动态创建聚合器

时间:2015-09-09 01:11:56

标签: google-cloud-platform google-cloud-dataflow

我们正在使用聚合器来处理处理期间的异常

public class BigTableWriter extends DoFn<String, Void> {

    private Aggregator<Integer, Integer> errorAggregator;
    public BigTableWriter(CloudBigtableOptions options) {
       errorAggregator = createAggregator("errors",new Sum.SumIntegerFn());
    }

    @Override
    public void processElement(DoFn<String, Void>.ProcessContext c){
        try {
          ....do work here
        }
        catch(Exception ex){
           errorAggregator.addValue(1);
        }

    }

}

我们希望使其更精细,而不是保留单个聚合器来收集错误。文档说聚合器通常在构造函数中创建。是否可以为catch块中的每个异常类型创建聚合器?例如,我们想要做类似的事情。

public class BigTableWriter extends DoFn<String, Void> {

    private Map<String, Aggregator<Integer, Integer> aggregatorMap;
    public BigTableWriter(CloudBigtableOptions options) {         
       aggregatorMap = new HashMap<>();
    }

    @Override
    public void processElement(DoFn<String, Void>.ProcessContext c){
        try {
          ....do work here
        }
        catch(Exception ex){
          aggregateException(ex.getCause().getMessage());
        }
    }

    public void aggregateException(String exceptionMessage) {
       Aggregator<Integer, Integer> aggregator = null;
       if(!aggregatorMap.containsKey(exceptionMessage){
          aggregator = createAggregator(exceptionMessage,new Sum.SumIntegerFn());
       }
       else {
          aggregator = aggregatorMap.get(exceptionMessage);
       }

         aggregator.put(exceptionMessage, aggregator);
    }

}

0 个答案:

没有答案