Question

我是pyspark的新手，无需多说。我有一个PySpark表，像这样：

+---+-------+-----+-------+
| id| ranges|score|    uom|
+---+-------+-----+-------+
|  1|    low|   20|percent|
|  1|verylow|   10|percent|
|  1|   high|   70|  bytes|
|  1| medium|   40|percent|
|  1|   high|   60|percent|
|  1|verylow|   10|percent|
|  1|   high|   70|percent|
+---+-------+-----+-------+

我想计算给定百分比为 0.95 的分数列的百分位值，我希望它也应返回相应的范围值。我尝试运行此查询：

results = spark.sql('select percentile_approx(score,0.95) as score, first(ranges)  from subset GROUP BY id')

我得到这样的结果：

+-----+--------------------+
|score|first(ranges, false)|
+-----+--------------------+
|   70|                 low|
+-----+--------------------+

它向我返回范围的第一个值，该值不正确，应该为“高”。如果我从查询中删除第一个（范围），则会出现错误：

> pyspark.sql.utils.AnalysisException: u"expression 'subset.`ranges`' is
> neither present in the group by, nor is it an aggregate function. Add
> to group by or wrap in first() (or first_value) if you don't care
> which value you get.;;\nAggregate [id#0L],
> [percentile_approx(score#2L, cast(0.95 as double), 10000, 0, 0) AS
> score#353L, ranges#1]\n+- SubqueryAlias subset\n   +- LogicalRDD
> [id#0L, ranges#1, score#2L, uom#3], false\n

Answer 1

这是因为您仅按ID分组。因此，通过使用第一个函数，您可以有效地从ranges列中选择一个随机值。

一种解决方案是创建第二个数据框，其中包含分数到范围的映射，然后最后将其重新连接到结果df中。

>>> df.registerTempTable("df") # Register first before selecting from 'df'
>>> map = spark.sql('select ranges, score from df')

>>> results = spark.sql('select percentile_approx(score,0.95) as score from subset GROUP BY id')

>>> results .registerTempTable("results ") 
>>> final_result = spark.sql('select r.score, m.ranges from results as r join map as m on r.score = m.score')

当percentile_approx基于groupby返回特定列的单个值时，如何选择另一列的对应值？

1 个答案: