Question

考虑到我在pyspark中有此数据框：

+--------+----------------+---------+---------+
|DeviceID| TimeStamp      |range    | zipcode |
+--------+----------------+---------+---------+
|   00236|11-03-2014 07:33|[4.5, 2] | 90041   |
|   00234|11-06-2014 05:55|[6.2, 8] | 90037   |
|   00234|11-06-2014 05:55|[5.6, 4] | 90037   |
|   00235|11-09-2014 05:33|[7.5, 6] | 90047   |
+--------+----------------+---------+---------+

当范围数组中的第一个值大于6时，如何编写一个保留行的脚本。输出应如下所示：

+--------+----------------+---------+---------+
|DeviceID| TimeStamp      |range    | zipcode |
+--------+----------------+---------+---------+
|   00234|11-06-2014 05:55|[6.2, 8] | 90037   |
|   00235|11-09-2014 05:33|[7.5, 6] | 90047   |
+--------+----------------+---------+---------+

我写了这个脚本：

import pyspark.sql.functions as f
df.filter(f.col("range")[0] > 6)

但我收到此错误：

AnalysisException: u"Can't extract value from range#12989: need struct type but got vector;"

Answer 1

df.filter(df.range[0]>6.0).show()

OR

df.withColumn("first_element", df.range[0])\
    .filter(col("first_element")>6.0).drop("first_element").show()

输出：

+--------+----------------+----------+-------+
|DeviceID|       TimeStamp|     range|zipcode|
+--------+----------------+----------+-------+
|   00235|11-09-2014 05:33|[7.5, 6.0]|  90047|
|   00234|11-06-2014 05:55|[6.2, 8.0]|  90037|
+--------+----------------+----------+-------+

如何基于列中数组的第一个值过滤pyspark数据帧？

1 个答案: