Question

我有一个PySpark数据框，想过滤上限和下限之间的行。通常，我只使用介于以下之间的过滤器：

import pandas as pd
from pyspark.sql import functions as F
... sql_context creation ...

pdfRaw=pd.DataFrame([{"vehicleID":'A', "Segment":'State Hwy', "speed":68.0},\
{"vehicleID":'B', "Segment":'State Hwy', "speed":76.0}])
dfRaw = sql_context.createDataFrame(pdfRaw).withColumn("vehicleID", "Segment", "speed")

dfRaw.show()

+-----------+------------+-----+
   vehicleID|     Segment|value|
+-----------+------------+-----+
|          A|   State Hwy| 68.0|
|          B|   State Hwy| 73.0|
+-----------+------------+-----+

dfRaw.filter(F.col("speed").between(70,75)).show()

+-----------+------------+-----+
   vehicleID|     Segment|value|
+-----------+------------+-----+
|          B|   State Hwy| 73.0|
+-----------+------------+-----+

但是我想在多个速度值之间进行过滤。

Speeds_Curious = {
[25,30],
[55,60],
[60,65],
[70,75]
}

我实际上想更进一步。过滤器的上限和下限取决于前一个数据帧的groupby结果。

df_RoadSegments.groupby('Segment')\
.agg(F.min('SpeedLimit').alias('minSpeed'),\
F.max('SpeedLimit').alias('maxSpeed'))\
.show()

+-----------+----------+----------+
     Segment|  minSpeed|  maxSpeed|
+-----------+----------+----------+
|      Urban|      25.0|      30.0|
|  State Hwy|      55.0|      60.0|
|I-State Hwy|      60.0|      65.0|
|I-State Hwy|      70.0|      75.0|
+-----------+----------+----------+

所以基本上，我想在一个值之间过滤一个数据帧，这些值可以作为不同数据帧上的列使用。

类似的东西：

dfLimits = df_RoadSegments.groupby('Segment')\
.agg(F.min('SpeedLimit').alias('minSpeed'),\ F.max('SpeedLimit').alias('maxSpeed'))

dfRaw.groupby('Segment')\
.filter(F.col("speed")\
.between(dfLimits.where(dfLimits.Segment=="State Hwy"(??)).select('minSpeed')),\
dfLimits.where(dfLimits.Segment=="State Hwy"(??)).select('maxSpeed'))))\
.show()

有什么想法吗？

Answer 1

以下方法将为您获取属于它们特定路段的min和max速度之间的所有车辆。

您可以加入两个数据框：

df_joined = dfRaw.join(dfLimits, on="Segment", how="left")
+---------+---------+-----+--------+--------+
|  Segment|vehicleID|speed|minSpeed|maxSpeed|
+---------+---------+-----+--------+--------+
|State Hwy|        A| 68.0|      55|      60|
|State Hwy|        B| 76.0|      55|      60|
+---------+---------+-----+--------+--------+

如果您想进一步确定速度是否在上述边界之间，则可以编写：

flag_df = df_joined.withColumn("flag", F.when((F.col("speed") > F.col("minSpeed")) & (F.col("speed") < F.col("minSpeed")), 1).otherwise(0))
flag_df.show()
+---------+---------+-----+--------+--------+----+
|  Segment|vehicleID|speed|minSpeed|maxSpeed|flag|
+---------+---------+-----+--------+--------+----+
|State Hwy|        A| 68.0|      55|      60|   0|
|State Hwy|        B| 76.0|      55|      60|   0|
+---------+---------+-----+--------+--------+----+

然后，您可以简单地过滤显示以下标记：

df_final = df.filter(F.col("flag") == 1)

PySpark过滤器之间-根据组提供上限和下限列表

1 个答案: