Question

给出以下示例数据框：

advertiser_id| name | amount    | total             |max_total_advertiser|
4061         |source1|-434.955284|-354882.75336200005| -355938.53950700007
4061         |source2|-594.012216|-355476.76557800005| -355938.53950700007
4061         |source3|-461.773929|-355938.53950700007| -355938.53950700007

我需要对金额和 max_total_advertiser 字段求和，以获取每行中正确的总价值。考虑到我需要按Advertiser_id划分的每个组此 total 值。（初始数据框中的总列不正确，这就是为什么我要正确计算的原因）

类似的东西应该是：

w = Window.partitionBy("advertiser_id").orderBy("advertiser_id")

df.withColumn("total_aux", when( lag("advertiser_id").over(w) == col("advertiser_id"), lag("total_aux").over(w) + col("amount") ).otherwise( col("max_total_advertiser") + col("amount") ))

此lag("total_aux")无法正常工作，因为尚未生成该列，这就是我要实现的目标，如果它是该组中的第一行，则对同一行中的列求和，如果不对先前获得的列求和当前金额字段的值。输出示例：

advertiser_id| name | amount    | total_aux             |
4061         |source1|-434.955284|-356373.494791    |
4061         |source2|-594.012216|-356967.507007    | 
4061         |source3|-461.773929|-357429.280936    |

谢谢。

Answer 1

我假设name是每个advertiser_id的不同值，因此您的数据集可以按name进行排序。我还假设max_total_advertiser的每个advertiser_id包含相同的值。如果不是其中一种，请添加评论。

您需要的是一个rangeBetween窗口，该窗口为您提供指定范围内的所有前排和后排。我们将使用Window.unboundedPreceding来汇总所有先前的值。

import pyspark.sql.functions as F
from pyspark.sql import Window

l = [
(4061, 'source1',-434.955284,-354882.75336200005, -355938.53950700007),
(4061, 'source2',-594.012216,-355476.76557800005, -345938.53950700007),
(4062, 'source1',-594.012216,-355476.76557800005, -5938.53950700007),
(4062, 'source2',-594.012216,-355476.76557800005, -5938.53950700007),
(4061, 'source3',-461.773929,-355938.53950700007, -355938.53950700007)
]

columns = ['advertiser_id','name' ,'amount', 'total', 'max_total_advertiser']

df=spark.createDataFrame(l, columns)

w = Window.partitionBy('advertiser_id').orderBy('name').rangeBetween(Window.unboundedPreceding, 0)

df = df.withColumn('total', F.sum('amount').over(w) + df.max_total_advertiser)
df.show()

输出：

+-------------+-------+-----------+-------------------+--------------------+ 
|advertiser_id|   name|     amount|              total|max_total_advertiser| 
+-------------+-------+-----------+-------------------+--------------------+ 
|         4062|source1|-594.012216|-6532.5517230000705|   -5938.53950700007| 
|         4062|source2|-594.012216| -7126.563939000071|   -5938.53950700007| 
|         4061|source1|-434.955284| -356373.4947910001| -355938.53950700007| 
|         4061|source2|-594.012216| -346967.5070070001| -345938.53950700007| 
|         4061|source3|-461.773929|-357429.28093600005| -355938.53950700007| 
+-------------+-------+-----------+-------------------+--------------------+

Answer 2

您可能正在寻找orderBy()函数。这样行吗？

from pyspark.sql.window import *

df.withColumn("cumulativeSum", sum(df("amount"))
             .over( Window.partitionBy("advertiser_id").orderBy("amount")))

PySpark的两个值的总和

2 个答案: