如果`DoFn`重置事件计时器

时间:2018-03-13 21:55:49

标签: google-cloud-dataflow apache-beam

我进行单元测试(使用TestStreamPAssertDoFn重置事件计时器。如果DoFn重置计时器,则测试将永久挂起,并且此行为似乎特定于事件域计时器。

这是光束测试设施中的错误还是预期的计时器行为?

这是一个玩具示例,我可以使用beam 2.3 SDK重现此行为。

static class KeyElements extends DoFn<String, KV<String, String>> {
    @ProcessElement
    public void processElement(ProcessContext context) {
        final String[] parts = context.element().split(":");
        if (parts.length == 2) {
            context.output(KV.of(parts[0], parts[1]));
        }
    }
}

static class TimerDoFn extends DoFn<KV<String, String>, KV<String, String>> {
    @TimerId("expiry")
    private final TimerSpec timerSpec = TimerSpecs.timer(TimeDomain.EVENT_TIME);

    @ProcessElement
    public void processElement(ProcessContext context, @TimerId("expiry") Timer timer) {
        timer.set(context.timestamp().plus(Duration.standardHours(1)));
        final KV<String, String> e = context.element();
        context.output(KV.of(e.getKey(), e.getValue() + "_output"));
    }

    @OnTimer("expiry")
    public void onExpiry(OnTimerContext context) {
        // do nothing
    }
}

@Rule
public TestPipeline p = TestPipeline.create();

@Test
public void testTimerDoFn() {
    TestStream<String> stream = TestStream
            .create(StringUtf8Coder.of())
            .addElements(
                    TimestampedValue.of("a:0", new Instant(0)),
                    TimestampedValue.of("a:1", new Instant(1)),
                    TimestampedValue.of("a:2", new Instant(2)),
                    TimestampedValue.of("a:3", new Instant(3)))
            .advanceWatermarkToInfinity();

    PCollection<KV<String, String>> result = p
            .apply(stream)
            .apply(ParDo.of(new KeyElements()))
            .apply(ParDo.of(new TimerDoFn()));

    PAssert.that(result).containsInAnyOrder(
            KV.of("a", "0_output"),
            KV.of("a", "1_output"),
            KV.of("a", "2_output"),
            KV.of("a", "3_output"));

    p.run();
}

如果输入元素为a:1, b:2, c:3, d:4,则上述测试将停止。

0 个答案:

没有答案