我有一个采用以下格式的数据集:
+-----------------------------+------------------------------------+------------+-----------------+
| timestamp |uuid |storyTopic |Type |
+-----------------------------+------------------------------------+------------+-----------------+
|2019-04-15 11:21:03.362 -0400|9e0f3d00-cff7-3b76-89df-0d11c0addc91|TRY1 |Draft |
|2019-04-15 11:21:06.547 -0400|null |TRY1 |Draft |
|2019-04-15 11:21:06.617 -0400|e142e9bc-6587-34e3-9042-959b624f5 |Trial12345 |Original |
|2019-04-15 11:21:08.196 -0400|null |TRY1 |Draft |
|2019-04-15 11:21:22.855 -0400|null |TRY1 |Draft |
|2019-04-15 11:23:36.108 -0400|null |TRY1 |Draft |
|2019-04-15 11:23:36.139 -0400|null |TRY1 |Draft |
|2019-04-15 11:23:50.311 -0400|null |TRY1 |Draft |
|2019-04-15 15:21:08.196 -0400|null |TRY1 |Draft |
|2019-04-15 15:21:12.617 -0400|e142e9bc-6587-34e3-9042-959b624f5 |Trial12345 |Original |
+-----------------------------+------------------------------------+------------+-----------------+
如果数据集的第一行的StoryTopic为TRY1
,则
1.找到 timeDifference 小于 5秒
的行,然后
2. uuid不为空
我尝试通过以下方式实现它:
//create time difference column
df.withColumn("diff", functions.unix_timestamp(functions.col("timestamp")));
//create a window function for time-range
WindowSpec w = Window.orderBy("diff").rangeBetween(1L, 5L);
df.withColumn("Type", functions.when(
(functions.rank().over(w)==1)
//checking if uuid is null
.and(functions.first("uuid").over(w).isNotNull()),
functions.first("storyTopic").over(w)).otherwise("storyTopic");
发生的事情是它忽略了uuid.isNotNull()
条件,而是从第二行而不是同时满足这两个条件的第三行获取数据。
如何确保functions.first()
选择uuid不为null的行?