我有一个SparkR DataFrame
,我希望为每个唯一value
获取模式(最常见)name
。我怎样才能做到这一点?似乎没有内置的mode
函数。 SparkR或PySpark解决方案都可以。
#Create DF
df <- data.frame(name = c("Thomas", "Thomas", "Thomas", "Bill", "Bill", "Bill"),
value = c(5, 5, 4, 3, 3, 7))
DF <- createDataFrame(df)
name | value
-----------------
Thomas | 5
Thomas | 5
Thomas | 4
Bill | 3
Bill | 3
Bill | 9
#What I want to get
name | mode(value)
-----------------
Thomas | 5
Bill | 3
答案 0 :(得分:4)
您可以使用.groupBy()
和window
方法的组合实现这一目标:
grouped = df.groupBy('name', 'value').count()
window = Window.partitionBy("name").orderBy(desc("count"))
grouped\
.withColumn('order', row_number().over(window))\
.where(col('order') == 1)\
.show()
输出:
+------+-----+-----+-----+
| name|value|count|order|
+------+-----+-----+-----+
| Bill| 3| 2| 1|
|Thomas| 5| 2| 1|
+------+-----+-----+-----+
答案 1 :(得分:0)
这是解决方案的SparkR版本:
grouped <- agg(groupBy(df, 'name', 'value'), count=count(df$value))
window <- orderBy(windowPartitionBy("name"), desc(grouped$count))
dfmode <- withColumn(grouped, 'order', over(row_number(), window))
dfmode <- filter(dfmode, dfmode$order==1)