Question

我正在使用PySpark的DataFrame部分来分析来自Apache Kafka的数据。我遇到了麻烦，需要一些帮助。

    from pyspark.sql import functions

    # selected_df is dataframe come from kafka use spark.readStream.format("kafka")...

    windowed_group_1 = selected_df.withWatermark("kafka_time", "10 minutes").groupBy(functions.window("kafka_time", "10 seconds", "5 seconds"))

    windowed_group_2 = selected_df.withWatermark("kafka_time", "10 minutes").groupBy(functions.window("kafka_time", "10 seconds", "5 seconds"))

两个groupby是否具有相同的窗口功能？它们具有相同的选项。

如果没有，我该怎么做？

windowed_group_1 == windowed_group_2

在此先感谢您的帮助。

Answer 1

也许这对我想要的有用，每当使用时间窗口时，窗口功能默认值就以1970-01-01T00：00：00作为参考帧。

from pyspark.sql import functions as func

a = labeled_df.groupBy(func.window("timestamp", "60 minute"), "proto").count().show(100, truncate=False)

b = labeled_df.groupBy(func.window("timestamp", "60 minute"), "proto").count().show(100, truncate=False)

结果a和b相同

a
+------------------------------------------+---------+-----+
|window                                    |proto    |count|
+------------------------------------------+---------+-----+
|[2010-06-13 08:00:00, 2010-06-13 09:00:00]|UDP      |1803 |
|[2010-06-13 02:00:00, 2010-06-13 03:00:00]|TCP      |22579|
|[2010-06-13 09:00:00, 2010-06-13 10:00:00]|TCP      |2637 |
|[2010-06-13 02:00:00, 2010-06-13 03:00:00]|IPv6-ICMP|453  |
|[2010-06-13 02:00:00, 2010-06-13 03:00:00]|UDP      |1183 |
|[2010-06-13 03:00:00, 2010-06-13 04:00:00]|UDP      |1467 |


b
+------------------------------------------+---------+-----+
|window                                    |proto    |count|
+------------------------------------------+---------+-----+
|[2010-06-13 08:00:00, 2010-06-13 09:00:00]|UDP      |1803 |
|[2010-06-13 02:00:00, 2010-06-13 03:00:00]|TCP      |22579|
|[2010-06-13 09:00:00, 2010-06-13 10:00:00]|TCP      |2637 |
|[2010-06-13 02:00:00, 2010-06-13 03:00:00]|IPv6-ICMP|453  |
|[2010-06-13 02:00:00, 2010-06-13 03:00:00]|UDP      |1183 |
|[2010-06-13 03:00:00, 2010-06-13 04:00:00]|UDP      |1467 |

数据框如何通过窗口功能获得相同的分组依据

1 个答案: