我为事件流创建了自定义触发和处理功能。
DataStream<DynamoDBRow> dynamoDBRows =
sensorEvents
.keyBy("id")
.window(GlobalWindows.create())
.trigger(new MyCustomTrigger())
.allowedLateness(Time.minutes(1)) # Note
.process(new MyCustomWindowProcessFunction());
我的触发器基于事件参数。收到事件结束信号后,将 MyCustomWindowProcessFunction()应用于窗口元素。
@Slf4j
public class MyCustomTrigger extends Trigger<SensorEvent, GlobalWindow> {
@Override
public TriggerResult onElement(SensorEvent element, long timestamp, GlobalWindow window, TriggerContext ctx) throws Exception {
if (element.isEventEnd() == true) {
return TriggerResult.FIRE_AND_PURGE;
}
return TriggerResult.CONTINUE;
}
@Override
public TriggerResult onProcessingTime(long time, GlobalWindow window, TriggerContext ctx) throws Exception {
return TriggerResult.CONTINUE;
}
@Override
public TriggerResult onEventTime(long time, GlobalWindow window, TriggerContext ctx) throws Exception {
return TriggerResult.CONTINUE;
}
@Override
public void clear(GlobalWindow window, TriggerContext ctx) throws Exception {}
}
可能很少有传感器数据,即使在触发之后也可能出现。因此,我添加了.allowedLateness(Time.minutes(1))
,以确保在处理时不会错过这些事件。
就我而言, allowedLateness 无法正常工作。
浏览文档后,我发现了
如何在GlobalWindow中包含 allowedLateness ?
注意:我还尝试设置环境时间特征
env.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime);
更新:20-02-2020
目前正在考虑以下方法。 (目前无法正常工作)
@Slf4j
public class JourneyTrigger extends Trigger<SensorEvent, GlobalWindow> {
private final long allowedLatenessMillis;
public JourneyTrigger(Time allowedLateness) {
this.allowedLatenessMillis = allowedLateness.toMilliseconds();
}
@Override
public TriggerResult onElement(SensorEvent element, long timestamp, GlobalWindow window, TriggerContext ctx) throws Exception {
if (element.isEventEnd() == true) {
log.info("Timer started with allowedLatenessMillis " + allowedLatenessMillis);
ctx.registerEventTimeTimer(System.currentTimeMillis() + allowedLatenessMillis);
}
return TriggerResult.CONTINUE;
}
@Override
public TriggerResult onEventTime(long time, GlobalWindow window, TriggerContext ctx) throws Exception {
log.info("onEvenTime called at "+System.currentTimeMillis() );
return TriggerResult.FIRE_AND_PURGE;
}
@Override
public TriggerResult onProcessingTime(long time, GlobalWindow window, TriggerContext ctx) throws Exception {
return TriggerResult.CONTINUE;
}
@Override
public void clear(GlobalWindow window, TriggerContext ctx) throws Exception {}
}
答案 0 :(得分:1)
最后,我可以使用以下自定义触发器满足要求。
import lombok.extern.slf4j.Slf4j;
import org.apache.flink.streaming.api.windowing.time.Time;
import org.apache.flink.streaming.api.windowing.triggers.Trigger;
import org.apache.flink.streaming.api.windowing.triggers.TriggerResult;
import org.apache.flink.streaming.api.windowing.windows.GlobalWindow;
@Slf4j
public class JourneyTrigger extends Trigger<SensorEvent, GlobalWindow> {
private final long allowedLatenessMillis;
public JourneyTrigger(Time allowedLateness) {
this.allowedLatenessMillis = allowedLateness.toMilliseconds();
}
@Override
public TriggerResult onElement(SensorEvent element, long timestamp, GlobalWindow window, TriggerContext ctx) throws Exception {
if (element.isEventEnd()==true) {
log.info("Timer started with allowedLatenessMillis " + allowedLatenessMillis);
ctx.registerProcessingTimeTimer(System.currentTimeMillis() + allowedLatenessMillis);
}
return TriggerResult.CONTINUE;
}
@Override
public TriggerResult onProcessingTime(long time, GlobalWindow window, TriggerContext ctx) throws Exception {
log.info("onProcessingTime called at "+System.currentTimeMillis() );
return TriggerResult.FIRE_AND_PURGE;
}
@Override
public TriggerResult onEventTime(long time, GlobalWindow window, TriggerContext ctx) throws Exception {
return TriggerResult.CONTINUE;
}
@Override
public void clear(GlobalWindow window, TriggerContext ctx) throws Exception {}
}
也在Driver.java
类中,设置环境时间特征
env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime);
答案 1 :(得分:0)
说实话,我看不出在这里使用GlobalWindow
的原因。您可以只使用KeyedProcessFunction
来达到与Trigger
相同的目的,基本上,它将把从事件开始到事件结束的所有元素收集到ListState
中然后,当您收到isEventEnd()==true
时,只需安排EventTime
计时器,该计时器将在一分钟后触发并发出ListState
内部收集的结果。