Question

我需要以下帮助。假设我有一个像下面这样的数据框。我希望根据“排名”列的顺序生成dom和attribute列的附录。

from pyspark.sql.functions import first
l =[( 1    ,'A', 10, 'size' ),
( 2        , 'B', 20, 'height' ),
( 3        , 'C', 30, 'weigth' )]

df = spark.createDataFrame(l, ['rank','dom', 'value', 'attribute'])

+----+---+-----+---------+
|rank|dom|value|attribute|
+----+---+-----+---------+
|   1|  A|   10|     size|
|   2|  B|   20|   height|
|   3|  C|   30|   weigth|
+---+---+-----+---------+

最终所需输出：

+----+-------+---------+--------------------+
|rank|    dom|avg_value|           attribute|
+----+-------+---------+--------------------+
|   1|      A|       10|                size|
|   2|    A,B|       20|        size, height|
|   3|  A,B,C|       30|size, height, weigth|
+----+-------+---------+--------------------+

这个想法是avg_price是按新属性组合分组的平均值。

Answer 1

您可以使用窗口功能并收集以前的属性值。

import pyspark.sql.functions as f
from pyspark.sql.window import Window

l =[( 1    ,'A', 10, 'size' ),
( 2        , 'B', 20, 'height' ),
( 3        , 'C', 30, 'weigth' )]

df = spark.createDataFrame(l, ['rank','dom', 'value', 'attribute'])

windowSpec = Window().orderBy('rank').rowsBetween(Window.unboundedPreceding, Window.currentRow)
df = df.withColumn('attribute',f.collect_list(f.col('attribute')).over(windowSpec))
df.show()

输出>

+----+---+-----+--------------------+
|rank|dom|value|           attribute|
+----+---+-----+--------------------+
|   1|  A|   10|              [size]|
|   2|  B|   20|      [size, height]|
|   3|  C|   30|[size, height, we...|
+----+---+-----+--------------------+

如果您有不同的数据组，则可以创建分区窗口

windowSpec = Window().partitionBy(PARTITION_COLUMN).orderBy('rank').rowsBetween(Window.unboundedPreceding, Window.currentRow)

在Pyspark中分组值

1 个答案: