Apache Beam无法在LocalRunner中与Session Windows一起使用

时间:2019-06-04 20:02:36

标签: apache-beam

我想在带有Kafka或SWS输入源的Apache Beam和LocalRunner上尝试Session Windows。 我无法让他们正常工作。窗户接缝不被处理 我有以下代码:

public class Test7Pipeline {

    private static final Logger LOG = LoggerFactory.getLogger(Test7Pipeline.class);

    public static void main(String[] args) throws ParseException {


        PipelineOptions options = PipelineOptionsFactory.create();


        Pipeline p = Pipeline.create(options);

        p.apply(KafkaIO.<Long, String>read()
                .withBootstrapServers("localhost:9092")
                .withTopic("test")
                .withKeyDeserializer(LongDeserializer.class)
                .withValueDeserializer(StringDeserializer.class)

                .updateConsumerProperties(ImmutableMap.of("auto.offset.reset", (Object)"earliest"))
                //.withMaxNumRecords(5)
                .withoutMetadata())
        .apply(Values.<String>create())

        .apply(
                "WindowIntoSessions",
                Window.<String>into(
                        Sessions.withGapDuration(Duration.standardMinutes(1)))
                .withTimestampCombiner(TimestampCombiner.END_OF_WINDOW))

        .apply("ExtractWords", ParDo.of(new DoFn<String, String>() {
            @ProcessElement
            public void processElement(ProcessContext c) {
                for (String word : c.element().split("/")) {
                    if (!word.isEmpty()) {
                        c.output(word);
                    }
                }
            }
        }))
        .apply(Count.<String>perElement())
        .apply("FormatResults", MapElements.via(new SimpleFunction<KV<String, Long>, String>() {
            @Override
            public String apply(KV<String, Long> input) {
                LOG.info("Value: " + input.getValue());
                return input.getKey() + ": " + input.getValue();
            }
        }))
        .apply("Log", ParDo.of(new FilterTextFn()));
        //.apply(TextIO.write().to("wordcounts"));


        p.run().waitUntilFinish();
    }


Kafka和SQS的行为相同。

0 个答案:

没有答案