Question

当我生成30天的数据时，我正在进行测试。当发送到SA作业时，所有输入都被删除了，但是根据事件排序刀片中的设置，我希望所有输入都能通过。

部分作业查询包含：

---------------all incoming events storage query
SELECT stream.*
INTO [iot-predict-SA2-ColdStorage]
FROM [iot-predict-SA2-input] stream TIMESTAMP BY stream.UtcTime

所以我的期望是把所有被推送到blob存储中的SA工作。

当我发送仅仅5个小时的事件时 - 输入被标记为延迟（预期）并进行处理。每个SS第一个标记区域显示过时事件输入，但没有输出（红色），第二个部分显示延迟处理事件。

完整查询

    WITH AlertsBasedOnMin
    AS (
        SELECT stream.SensorGuid
            ,stream.Value
            ,stream.SensorName
            ,ref.AggregationTypeFlag
            ,ref.MinThreshold AS threshold
            ,ref.Count
            ,CASE 
                WHEN (ref.MinThreshold > stream.Value)
                    THEN 1
                ELSE 0
                END AS isAlert
        FROM [iot-predict-SA2-input] stream TIMESTAMP BY stream.UtcTime
        JOIN [iot-predict-SA2-referenceBlob] ref ON ref.SensorGuid = stream.SensorGuid
        WHERE ref.AggregationTypeFlag = 8
        )
        ,AlertsBasedOnMax
    AS (
        SELECT stream.SensorGuid
            ,stream.Value
            ,stream.SensorName
            ,ref.AggregationTypeFlag
            ,ref.MaxThreshold AS threshold
            ,ref.Count
            ,CASE 
                WHEN (ref.MaxThreshold < stream.Value)
                    THEN 1
                ELSE 0
                END AS isAlert
        FROM [iot-predict-SA2-input] stream TIMESTAMP BY stream.UtcTime
        JOIN [iot-predict-SA2-referenceBlob] ref ON ref.SensorGuid = stream.SensorGuid
        WHERE ref.AggregationTypeFlag = 16
        )
        ,alertMinMaxUnion
    AS (
        SELECT *
        FROM AlertsBasedOnMin

        UNION ALL

        SELECT *
        FROM AlertsBasedOnMax
        )
        ,alertMimMaxComputed
    AS (
        SELECT SUM(alertMinMaxUnion.isAlert) AS EventCount
            ,alertMinMaxUnion.SensorGuid AS SensorGuid
            ,alertMinMaxUnion.SensorName
        FROM alertMinMaxUnion
        GROUP BY HoppingWindow(Duration(minute, 1), Hop(second, 30))
            ,alertMinMaxUnion.SensorGuid
            ,alertMinMaxUnion.Count
            ,alertMinMaxUnion.AggregationTypeFlag
            ,alertMinMaxUnion.SensorName
        HAVING SUM(alertMinMaxUnion.isAlert) > alertMinMaxUnion.Count
        )
        ,alertsMimMaxComputedMergedWithReference
    AS (
        SELECT System.TIMESTAMP [TimeStampUtc]
            ,computed.EventCount
            ,0 AS SumValue
            ,0 AS AvgValue
            ,0 AS StdDevValue
            ,computed.SensorGuid
            ,computed.SensorName
            ,ref.MinThreshold
            ,ref.MaxThreshold
            ,ref.TimeFrameInSeconds
            ,ref.Count
            ,ref.GatewayGuid
            ,ref.SensorType
            ,ref.AggregationType
            ,ref.AggregationTypeFlag
            ,ref.EmailList
            ,ref.PhoneNumberList
        FROM alertMimMaxComputed computed
        JOIN [iot-predict-SA2-referenceBlob] ref ON ref.SensorGuid = computed.SensorGuid
        )
        ,alertsAggregatedByFunction
    AS (
        SELECT Count(1) AS eventCount
            ,stream.SensorGuid AS SensorGuid
            ,stream.SensorName
            ,ref.[Count] AS TriggerThreshold
            ,SUM(stream.Value) AS SumValue
            ,AVG(stream.Value) AS AvgValue
            ,STDEV(stream.Value) AS StdDevValue
            ,ref.AggregationTypeFlag AS flag
        FROM [iot-predict-SA2-input] stream TIMESTAMP BY stream.UtcTime
        JOIN [iot-predict-SA2-referenceBlob] ref ON ref.SensorGuid = stream.SensorGuid
        GROUP BY HoppingWindow(Duration(minute, 1), Hop(second, 30))
            ,ref.AggregationTypeFlag
            ,ref.[Count]
            ,ref.MaxThreshold
            ,ref.MinThreshold
            ,stream.SensorGuid
            ,stream.SensorName
        HAVING
            --as this is alert then this factor will be relevant to all of the aggregated queries
            Count(1) >= ref.[Count]
            AND (
                --average
                (
                    ref.AggregationTypeFlag = 1
                    AND (
                        AVG(stream.Value) >= ref.MaxThreshold
                        OR AVG(stream.Value) <= ref.MinThreshold
                        )
                    )
                --sum
                OR (
                    ref.AggregationTypeFlag = 2
                    AND (
                        SUM(stream.Value) >= ref.MaxThreshold
                        OR Sum(stream.Value) <= ref.MinThreshold
                        )
                    )
                --stdev 
                OR (
                    ref.AggregationTypeFlag = 4
                    AND (
                        STDEV(stream.Value) >= ref.MaxThreshold
                        OR STDEV(stream.Value) <= ref.MinThreshold
                        )
                    )
                )
        )
        ,alertsAggregatedByFunctionMergedWithReference
    AS (
        SELECT System.TIMESTAMP [TimeStampUtc]
            ,0 AS EventCount
            ,computed.SumValue
            ,computed.AvgValue
            ,computed.StdDevValue
            ,computed.SensorGuid
            ,computed.SensorName
            ,ref.MinThreshold
            ,ref.MaxThreshold
            ,ref.TimeFrameInSeconds
            ,ref.Count
            ,ref.GatewayGuid
            ,ref.SensorType
            ,ref.AggregationType
            ,ref.AggregationTypeFlag
            ,ref.EmailList
            ,ref.PhoneNumberList
        FROM alertsAggregatedByFunction computed
        JOIN [iot-predict-SA2-referenceBlob] ref ON ref.SensorGuid = computed.SensorGuid
        )
        ,allAlertsUnioned
    AS (
        SELECT *
        FROM alertsAggregatedByFunctionMergedWithReference

        UNION ALL

        SELECT *
        FROM alertsMimMaxComputedMergedWithReference
        )


    ---------------alerts storage query 
    SELECT *
    INTO [iot-predict-SA2-Alerts-ColdStorage]
    FROM allAlertsUnioned

    ---------------alerts to alert events query
    SELECT *
    INTO [iot-predict-SA2-Alerts-EventStream]
    FROM allAlertsUnioned


    ---------------alerts to stream query
    SELECT *
    INTO [iot-predict-SA2-TSI-EventStream]
    FROM allAlertsUnioned


    ---------------all incoming events storage query
    SELECT stream.*
    INTO [iot-predict-SA2-ColdStorage]
    FROM [iot-predict-SA2-input] stream TIMESTAMP BY stream.UtcTime


    ---------------all incoming events to time insights query
    SELECT stream.*
    INTO [iot-predict-SA2-TSI-AlertStream]
    FROM [iot-predict-SA2-input] stream TIMESTAMP BY stream.UtcTime

Answer 1

由于您使用的是“TIMESTAMP BY”，因此Stream Analytics作业事件排序设置正在生效。请检查您的工作的“事件订购”设置，特别是在两个以下：

迟到的事件 - 迟到的限制在0秒到21天之间。
处理其他事件 - 错误处理策略，将应用程序时间丢弃或调整为系统时钟时间。

我想，最有可能的是，您的迟到限制超过5小时，因此可以处理这些5小时的事件。

您可能已经从上面弄清楚，Stream Analytics作业只能处理最晚21天的“旧”事件。要解决此限制，您可以考虑以下选项之一：

删除TIMESTAMP BY，然后所有窗口聚合将使用入队时间。根据您的查询逻辑，这可能会生成错误的结果。
选择“adjust”作为错误处理策略。同样，根据您的查询逻辑，这可能会生成错误的结果。
使用DATEADD（）函数将应用程序时间（stream.UtcTime）转换为更多时间，例如TIMESTAMP BY DATEADD（day，10，UtcTime）。当这是一次性任务时，这很有效，并且您知道事件的时间范围。
使用批处理作业（Stream Analytics外部）处理30天前的数据。

Answer 2

在与来自MS的人聊天之后，我的测试必须有一个额外的步骤才能完成。

要处理延迟事件，无论后期事件设置如何，我们都需要以某种方式启动此作业，后期事件被视为在作业启动时发送，因此在此特定情况下，我们必须使用启动SA作业自定义开始日期并在30天前设置。

SA事件的后期事件

完整查询

2 个答案: