Flink-使用重新平衡影响事件时间处理

时间:2020-05-10 15:06:39

标签: apache-flink flink-streaming

在分配水印功能之前使用重新平衡会影响事件时间处理

Configuration config = new Configuration();
            StreamExecutionEnvironment see = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(config);

            see.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);


            DataStream<String> stream = see
                    .addSource(new FlinkKafkaConsumer<>("testTopic3", new SimpleStringSchema(), properties));


            stream.rebalance()
            .map(event->{
                JSONObject jsonEvent = new JSONObject(event);
                Event formedEvent = new Event();
                formedEvent.setTimestamp(Long.parseLong(jsonEvent.getString("time")));
                System.out.println("formed event : "+formedEvent.getTimestamp());
                return formedEvent;
            })
            .assignTimestampsAndWatermarks(new TimestampExtractor())
            .keyBy(event->{
                return "1";
            }).window(SlidingEventTimeWindows.of(Time.seconds(5), Time.seconds(5)))
            .trigger(new TriggerFunc())
            .process(new ProcessWindowFunction());

            see.execute();

使用重新平衡时,事件时间触发器不会触发。删除后,它似乎可以正常工作。有什么理由不起作用吗?

以下似乎有效:

Configuration config = new Configuration();
            StreamExecutionEnvironment see = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(config);

            see.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);


            DataStream<String> stream = see
                    .addSource(new FlinkKafkaConsumer<>("testTopic3", new SimpleStringSchema(), properties));


            stream
            .map(event->{
                JSONObject jsonEvent = new JSONObject(event);
                Event formedEvent = new Event();
                formedEvent.setTimestamp(Long.parseLong(jsonEvent.getString("time")));
                System.out.println("formed event : "+formedEvent.getTimestamp());
                return formedEvent;
            })
            .assignTimestampsAndWatermarks(new TimestampExtractor())
.rebalance()
            .keyBy(event->{
                return "1";
            }).window(SlidingEventTimeWindows.of(Time.seconds(5), Time.seconds(5)))
            .trigger(new TriggerFunc())
            .process(new ProcessWindowFunction());

            see.execute();

添加我的触发功能。我的触发功能执行不了什么。只是打印一条语句

public class TriggerFunc  extends Trigger<Event, TimeWindow>{

    private static final long serialVersionUID = 1L;
    @Override
    public TriggerResult onProcessingTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {
        System.out.println("On processing time purging : " + window.getStart());
        return TriggerResult.FIRE;
    }

    @Override
    public TriggerResult onEventTime(long time, TimeWindow window, TriggerContext ctx) throws Exception {
        System.out.println("On eventTime time");
        return TriggerResult.FIRE_AND_PURGE;
    }

    @Override
    public void clear(TimeWindow window, TriggerContext ctx) throws Exception {

    }

    @Override
    public TriggerResult onElement(ThresholdEvent element, long timestamp, TimeWindow window, TriggerContext ctx)
            throws Exception {
        System.out.println("On element");
        return TriggerResult.CONTINUE;
    }
}

0 个答案:

没有答案