Question

我找到了以下代码，用于从由unique_id分组的数据帧中选择n行。

import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions.row_number

val window = Window.partitionBy("userId").orderBy($"rating".desc)

dataframe.withColumn("r", row_number.over(window)).where($"r" <= n)

我尝试了以下内容：

from pyspark.sql.functions import row_number, desc
from pyspark.sql.window import Window

w = Window.partitionBy(post_tags.EntityID).orderBy(post_tags.Weight)
newdata=post_tags.withColumn("r", row_number.over(w)).where("r" <= 3)

我收到以下错误：

AttributeError: 'function' object has no attribute 'over'

请帮助我。

Answer 1

我找到了答案：

from pyspark.sql.window import Window
from pyspark.sql.functions import rank, col

window = Window.partitionBy(df['user_id']).orderBy(df['score'].desc())

df.select('*', rank().over(window).alias('rank')) 
  .filter(col('rank') <= 2) 
  .show()

致@mtoto回答https://stackoverflow.com/a/38398563/5165377

将Scala代码转换为PySpark

1 个答案: