Question

我试图像下面那样进行流媒体处理，

从来源收集传感器事件
收集为设备定义的规则事件（传输传感器事件）
可能已使用不同设备的聚合过程的窗口信息定义规则
当具有在流中看到的窗口信息的设备的规则然后创建窗口（翻滚）
如果新规则没有窗口信息，则删除窗口并处理没有窗口功能。

我将此作为参考：https://jsfiddle.net/n4v63g3L/6/

我的流媒体代码如下：

mappedDataSource 
    .connect(mappedRuleStream) 
    .keyBy(..deviceId..) 
    .process(new RuleProcessorFunction()) 
    .windowAll(new CustomTimeWindowing()) 
    .apply(new AllWindowFunction<ProcessedEvent, Object, TimeWindow>() { 

      @Override 
      public void apply(TimeWindow window, Iterable<ProcessedEvent> values, Collector out) throws Exception { 
        System.out.println("hello"); 
      } 
    });

RuleProcessorFunction是：

public class RuleProcessorFunction extends CoProcessFunction<SensorEvent, RuleEvent, ProcessedEvent> { 

  private transient ValueState<Tuple2<SensorEvent, RuleEvent>> state; 

  @Override 
  public void processElement1(SensorEvent value, Context ctx, Collector<ProcessedEvent> out) throws Exception { 
    System.out.println("process element device id : " + value.deviceId); 
    System.out.println("process element solution id : " + value.solutionId); 
    state.update(Tuple2.of(value, null)); 

    RuleEvent rule = state.value().f1; 

    // execute if there is a defined rule on incoming event 
  } 

  @Override 
  public void processElement2(RuleEvent value, Context ctx, Collector<ProcessedEvent> out) throws Exception { 
    System.out.println("rule stream element solId :" + value.solutionId + " devId : " + value.deviceId); 
    state.value().f1 = value; 
    // store rule in memory 
    // processed event is gonna be stored window information and downstream is window assignment 
    ProcessedEvent processedEvent = new ProcessedEvent(); 
    processedEvent.deviceId = value.deviceId; 
    processedEvent.solutionId = value.solutionId; 
    processedEvent.windowInfo = value.window; 
    processedEvent.ruleId = value.ruleId; 

    out.collect(processedEvent); 

  } 

  @Override 
  public void open(Configuration parameters) throws Exception { 
    ValueStateDescriptor<Tuple2<SensorEvent, RuleEvent>> stateDescriptor = 
            new ValueStateDescriptor<>("processor", TypeInformation.of(new TypeHint<Tuple2<SensorEvent, RuleEvent>>() { 
            })); 
    state = getRuntimeContext().getState(stateDescriptor); 

  } 

  @Override 
  public void onTimer(long timestamp, OnTimerContext ctx, Collector<ProcessedEvent> out) throws Exception { 
    // rule triggers 
  } 
}

CustomWindowAssigner是：

public class CustomTimeWindowing extends TumblingEventTimeWindows { 
  public CustomTimeWindowing() { 
    super(1, 0); 
  } 

  @Override 
  public Collection<TimeWindow> assignWindows(Object element, long timestamp, WindowAssignerContext context) { 
    System.out.println("creating window : "); 
    ProcessedEvent processedEvent = (ProcessedEvent) element; 
    int windowInfo = processedEvent.windowInfo; 
    System.out.println("creating window  rule : " + processedEvent.ruleId); 
    long size = windowInfo * 1000; 
    System.out.println("window info in milisecond :" + size); 
    long start = timestamp - (timestamp % size); 
    long end = start + size; 
    return Collections.singletonList(new TimeWindow(start, end)); 
  } 
}

当ruleEvent出现时，我会添加有关窗口信息的元数据并添加到收集器中以保持流式传输。但如果我在SensorEvent的processElement1中执行此操作，则会再次调用windowAssigner并更改窗口。我希望它在新的/更改的窗口信息出现时进入。

创建这种结构的正确方法是什么？手动管理Windows还是使用这种自定义窗口分配器？

另一个参考：https://techblog.king.com/rbea-scalable-real-time-analytics-king/

如何在process（）运算符之后动态分配窗口

0 个答案: