Question

我有一个简单的数据集，如下所示。

|          id|         name|             country|          languages|    
|1           |          Bob|                 USA|            Spanish|
|2           |     Angelina|              France|               null|                
|3           |         Carl|              Brazil|               null|              
|4           |         John|           Australia|            English|        
|5           |         Anne|               Nepal|               null|

我试图使用pyspark.sql.window在languages中使用最后一个非空值来归零空值，以在某些行上创建一个窗口，但不会发生任何事情。应该填充空值的列temp_filled_spark,保持不变，即原始languages列的副本。

from pyspark.sql import Window
from pyspark.sql.functions import last

window = Window.partitionBy('name').orderBy('country').rowsBetween(-sys.maxsize, 0)
filled_column = last(df['languages'], ignorenulls=True).over(window)

df = df.withColumn('temp_filled_spark', filled_column)

df.orderBy('name', 'country').show(100)

我希望输出列为：

|temp_filled_spark|    
|     Spanish|
|     Spanish|                
|     Spanish|              
|     English|
|     English|

有人可以帮忙指出错误吗？

Answer 1

我们可以创建窗口，将整个数据帧视为一个分区，

from pyspark.sql import functions as F
>>> df1.show()
+---+--------+---------+---------+
| id|    name|  country|languages|
+---+--------+---------+---------+
|  1|     Bob|      USA|  Spanish|
|  2|Angelina|   France|     null|
|  3|    Carl|   Brazil|     null|
|  4|    John|Australia|  English|
|  5|    Anne|    Nepal|     null|
+---+--------+---------+---------+

>>> w = Window.partitionBy(F.lit(1)).orderBy(F.lit(1)).rowsBetween(-sys.maxsize, 0)
>>> df1.select("*",F.last('languages',True).over(w).alias('newcol')).show()
+---+--------+---------+---------+-------+
| id|    name|  country|languages| newcol|
+---+--------+---------+---------+-------+
|  1|     Bob|      USA|  Spanish|Spanish|
|  2|Angelina|   France|     null|Spanish|
|  3|    Carl|   Brazil|     null|Spanish|
|  4|    John|Australia|  English|English|
|  5|    Anne|    Nepal|     null|English|
+---+--------+---------+---------+-------+

希望这有帮助。！

PySpark中的前向填充缺失数据不起作用

1 个答案: