Question

使用WordCount.java示例测试指标。管道完成后，PrintMetricsFn函数（请参见下文）可以正确打印度量，但是在处理管道时调用该函数会失败。 DirectRunner和FlinkRunner都抛出异常（空输入），而SparkRunner仅打印标题行（类似于“ *度量值* ”）。

 private static void PrintMetricsFn(final MetricsFilter inputFilter) {
    final MetricResults metrics = result.metrics();
    final MetricQueryResults returns = metrics.queryMetrics(inputFilter);
    final Iterable<MetricResult<Long>> counters = returns.getCounters();

    System.out.println("*** Metric Values ***");
    for(final MetricResult<Long> s: counters){
        System.out.println(s.getName().getNamespace() + ":" +
                s.getName().getName() + ":" + s.getStep() + ": count=" +
                s.getAttempted()) ;
    }
    final Iterable<MetricResult<DistributionResult>> dist = returns.getDistributions();
    for (final MetricResult<DistributionResult> d: dist) {
        System.out.println(d.getName().getNamespace() + ":" +
                d.getName().getName() + ":" + d.getStep() + ": sum=" +
                d.getAttempted().getSum() + ", count=" + 
                d.getAttempted().getCount() + ", min=" +
                d.getAttempted().getMin() + ", max=" + 
                d.getAttempted().getMax() + ", mean=" +
                d.getAttempted().getMean());
    }
    final Iterable<MetricResult<GaugeResult>> gauge = returns.getGauges();
    for (final MetricResult<GaugeResult> g: gauge) {
        System.out.println(g.getName().getNamespace() + ":" +
                g.getName().getName() + ":" + g.getStep() + ": gauge=" +
                g.getAttempted().getValue());
    }
    return;
}

以下是外部变量的声明：

private static PipelineResult result;
private static MetricQueryResults returns;
private static MetricsFilter mFilter;

这是设置mFilter和结果变量的地方：

mFilter = MetricsFilter.builder()
        .addNameFilter(MetricNameFilter.named("ExtractWordsFn", "emptyLines"))
        .addNameFilter(MetricNameFilter.named("ExtractWordsFn", "lineLenDistro"))
        .addNameFilter(MetricNameFilter.named("ExtractWordsFn", "maxLineLen"))
        .build();

result = p.run();

最后，这是没有成功的调用PrintMetricsFN的函数：

@ProcessElement
public void processElement(@Element final String element, final OutputReceiver<String> receiver) {
  lineLen = element.length();
  lineLenDist.update(lineLen);
  if (element.trim().isEmpty()) {
    emptyLines.inc();
  }

  if (lineLen > localMax) {
      localMax = lineLen;
      maxLineLen.set(localMax);
  }

  if((lineCnt++ % 10) == 0) {
      PrintMetricsFn(mFilter);
  }

  // Split the line into words.
  final String[] words = element.split(ExampleUtils.TOKENIZER_PATTERN, -1);

  // Output each word encountered into the output PCollection.
  for (final String word : words) {
    if (!word.isEmpty()) {
      receiver.output(word);
    }
  }
}

为了完整起见，这是我从DirectRunner获得的错误：

Caused by: java.lang.NullPointerException
at org.apache.beam.examples.WordCount.PrintMetricsFn(WordCount.java:158)
at org.apache.beam.examples.WordCount.access$1(WordCount.java:157)
at org.apache.beam.examples.WordCount$ExtractWordsFn.processElement(WordCount.java:132)

Apache Beam：在管道处理期间访问指标

0 个答案: