Question

假设我正在尝试对一些包含对（a和b值）的示例数据进行一些统计。有些对存在多次，而另一些则不存在。

spark.createDataFrame([
    Row(a=5, b=10), Row(a=5, b=10), Row(a=5, b=10),
    Row(a=6, b=10), Row(a=6, b=10), Row(a=6, b=10), Row(a=6, b=10), Row(a=6, b=10), Row(a=6, b=10),
    Row(a=5, b=11), Row(a=5, b=11),
    Row(a=6, b=12), Row(a=6, b=12), Row(a=6, b=12), Row(a=6, b=12),
    Row(a=5, b=5), Row(a=5, b=5), Row(a=5, b=5), Row(a=5, b=5), Row(a=5, b=5), Row(a=5, b=5), Row(a=5, b=5),
]).registerTempTable('mydata')

首先，我只是在计算每对存在的频率：

spark.sql('''
SELECT a, b,
    COUNT(*) as count
FROM mydata AS o
GROUP BY a, b
''').show()

输出：

+---+---+-----+
|  a|  b|count|
+---+---+-----+
|  6| 12|    4|
|  5|  5|    7|
|  6| 10|    6|
|  5| 10|    3|
|  5| 11|    2|
+---+---+-----+

现在，我想添加一个额外的列，其中包含一个对存在的频率相对于具有相同值的对总数的百分比。为此，我尝试添加一个相关的子查询来计算总数：

spark.sql('''
SELECT a, b,
    COUNT(*) as count,
    (COUNT(*) / (
        SELECT COUNT(*) FROM mydata AS i WHERE o.a = i.a
    )) as percentage
FROM mydata AS o
GROUP BY a, b
''').show()

我期望的是

+---+---+-----+----------+
|  a|  b|count|percentage|
+---+---+-----+----------+
|  6| 12|    4|       0.4|  --> 10 pairs exist with a=6 --> 4/10 = 0.4
|  5|  5|    7|    0.5833|  --> 12 pairs exist with a=5 --> 7/12  =0.5833
|  6| 10|    6|       0.6|  --> ...
|  5| 10|    3|      0.25|
|  5| 11|    2|    0.1666|
+---+---+-----+----------+

我得到的是什么

py4j.protocol.Py4JJavaError: An error occurred while calling o371.showString.
: org.apache.spark.sql.catalyst.errors.package$TreeNodeException: Binding attribute, tree: count(1)#382L
    at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:56)
    at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:91)
    at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:90)
[...]
Caused by: java.lang.RuntimeException: Couldn't find count(1)#382L in [a#305L,b#306L,count(1)#379L]
    at scala.sys.package$.error(package.scala:27)
    at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1$$anonfun$applyOrElse$1.apply(BoundAttribute.scala:97)
    at org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1$$anonfun$applyOrElse$1.apply(BoundAttribute.scala:91)
    at org.apache.spark.sql.catalyst.errors.package$.attachTree(package.scala:52)
    ... 80 more

这听起来有点令人困惑-pyspark以某种方式想要访问内部联接的计数？

我的子查询语法有问题吗？

Answer 1

从第一个表中，您可以使用窗口函数来计算百分比； sum(count) over (partition by a)将count乘以a，结果的长度不减少，这使您可以直接用另一列除：

spark.sql('''
    SELECT a, b,
        COUNT(*) as count
    FROM mydata AS o
    GROUP BY a, b
''').registerTempTable('count')

spark.sql('''
    SELECT *, 
           count / sum(count) over (partition by a) as percentage 
    FROM count
''').show()
+---+---+-----+-------------------+
|  a|  b|count|         percentage|
+---+---+-----+-------------------+
|  6| 12|    4|                0.4|
|  6| 10|    6|                0.6|
|  5|  5|    7| 0.5833333333333334|
|  5| 10|    3|               0.25|
|  5| 11|    2|0.16666666666666666|
+---+---+-----+-------------------+

分组表达式上的相关子查询-TreeNodeException：绑定属性，树：count（1）＃382L

1 个答案: