我想在带有Kafka或SWS输入源的Apache Beam和LocalRunner上尝试Session Windows。 我无法让他们正常工作。窗户接缝不被处理 我有以下代码:
public class Test7Pipeline {
private static final Logger LOG = LoggerFactory.getLogger(Test7Pipeline.class);
public static void main(String[] args) throws ParseException {
PipelineOptions options = PipelineOptionsFactory.create();
Pipeline p = Pipeline.create(options);
p.apply(KafkaIO.<Long, String>read()
.withBootstrapServers("localhost:9092")
.withTopic("test")
.withKeyDeserializer(LongDeserializer.class)
.withValueDeserializer(StringDeserializer.class)
.updateConsumerProperties(ImmutableMap.of("auto.offset.reset", (Object)"earliest"))
//.withMaxNumRecords(5)
.withoutMetadata())
.apply(Values.<String>create())
.apply(
"WindowIntoSessions",
Window.<String>into(
Sessions.withGapDuration(Duration.standardMinutes(1)))
.withTimestampCombiner(TimestampCombiner.END_OF_WINDOW))
.apply("ExtractWords", ParDo.of(new DoFn<String, String>() {
@ProcessElement
public void processElement(ProcessContext c) {
for (String word : c.element().split("/")) {
if (!word.isEmpty()) {
c.output(word);
}
}
}
}))
.apply(Count.<String>perElement())
.apply("FormatResults", MapElements.via(new SimpleFunction<KV<String, Long>, String>() {
@Override
public String apply(KV<String, Long> input) {
LOG.info("Value: " + input.getValue());
return input.getKey() + ": " + input.getValue();
}
}))
.apply("Log", ParDo.of(new FilterTextFn()));
//.apply(TextIO.write().to("wordcounts"));
p.run().waitUntilFinish();
}
Kafka和SQS的行为相同。