Question

我的火花数据看起来像-

area           product           score          
a               aa                 .39
a               bb                 .03
a               cc                 1.1
a               dd                 .5
b               ee                 .02
b               aa                 1.2
b               mm                  .5
b               bb                 1.3

我想根据得分变量对前3名产品区域进行明智排名。我的最终输出应该是

area           product           score          rank
a               cc                 1.1            1
a               dd                 .5             2 
a               a                  .39            3
b               bb                 1.3            1 
b               aa                 1.2            2
b               mm                  .5            3

如何在PySpark中做到这一点？

我到目前为止已经完成了-

from pyspark.sql import Window
import pyspark.sql.functions as psf
wA = Window.orderBy(psf.desc("score"))
df = df.withColumn(
    "rank", 
    psf.dense_rank().over(wA))

但不能为我工作。

Answer 1

通过area进行分区并使用过滤器rank<=3将给出结果

import pyspark.sql.functions as psf
from pyspark.sql import SparkSession
from pyspark.sql.window import Window

spark = SparkSession.builder.appName("Test").master("local[*]") \
    .getOrCreate()
df = spark.createDataFrame([('a', 'aa', .39),
                            ('a', 'bb', .03),
                            ('a', 'cc', 1.1),
                            ('a', 'dd', .5),
                            ('b', 'ee', .02),
                            ('b', 'aa', 1.2),
                            ('b', 'mm', .5),
                            ('b', 'bb', 1.3)],
                           ['area', 'product', 'score'])

wA = Window.partitionBy("area").orderBy(psf.desc("score"))
df = df.withColumn("rank",
                   psf.dense_rank().over(wA))
df.filter("rank<=3").show()

在PySpark中得分明智的排名

1 个答案: