Question

我尝试将按其分钟时间戳键入的60秒数据聚合，最多延迟30秒。

DataStream<OHLChelp> ohlcAggStream = stockStream.assignTimestampsAndWatermarks(new TimestampExtractor(Time.seconds(30))).map(new mapStockToOhlcHelp()).keyBy((KeySelector<OHLChelp, Long>) o -> o.getMinTime())
            .timeWindow(Time.seconds(60))
            .reduce(new aggregateOHLC());
//map complex object to simpler one
DataStream<OHLCmodel> ohlcStremAggregated = ohlcAggStream.map(new mapOHLCredToOHLCfin());
//log ohlc stream
ohlcStreamAggregated.writeAsText(outLogPath);

我收到了数据。水印和时间戳正在设定。似乎，聚合数据永远不会发送到ohlcStreamAggregated，因此它们不会被记录。

public TimestampExtractor(Time maxDelayInterval) {
        if (maxDelayInterval.toMilliseconds() < 0) {
            throw new RuntimeException("This parameter must be positive or 0.);
        }
        this.maxDelayInterval = maxDelayInterval.toMilliseconds() / 1000;
        this.currentMaxTimestamp = Long.MIN_VALUE + this.maxDelayInterval;
    }

@Override
public final Watermark getCurrentWatermark() {
        // set maximum delay 30 seconds
        long potentialWM = currentMaxTimestamp - maxDelayInterval;
        if (potentialWM > lastEmittedWM) {
            lastEmittedWM = potentialWM;
        }
        return new Watermark(lastEmittedWM);
    }
@Override
public final long extractTimestamp(StockTrade stockTrade, long previousElementTimestamp) {
        BigDecimal bd = new BigDecimal(stockTrade.getTime());
        long timestamp = bd.longValue();
        //set the maximum seen timestamp so far
        if (timestamp > currentMaxTimestamp) {
            currentMaxTimestamp = timestamp;
        }
        return timestamp;
    }

我使用this example作为模板。

Answer 1

如果您可以分享整个事情（可能是一个要点），那么诊断您的应用程序会更容易，但是，你做过：

将时间特征设置为事件时间（docs）？
在流执行环境中调用execute？

此外，您的时间戳提取器可能会更简单。更像是这样：

public static class TimestampExtractor extends BoundedOutOfOrdernessTimestampExtractor<StockTrade> {
    public TimestampExtractor() {
        super(Time.seconds(30));
    }

    @Override
    public long extractTimestamp(StockTrade trade) {
        return trade.getTime();
    }
}

Apache Flink。用水印加窗

1 个答案: