Question

我正在尝试对包含数组的字段求和，

a = sc.parallelize([("a", [1,1,1]),
                    ("a", [2,2])])

a = a.toDF(["g", "arr_val"])

a.registerTempTable('a')

sql = """
select 
aggregate(arr_val, 0, (acc, x) -> acc + x) as sum
from a
"""

spark.sql(sql).show()

但是我遇到了以下错误：

An error occurred while calling o24.sql.
: org.apache.spark.sql.AnalysisException: cannot resolve 'aggregate(a.`arr_val`, 0, lambdafunction((CAST(namedlambdavariable() AS BIGINT) + namedlambdavariable()), namedlambdavariable(), namedlambdavariable()), lambdafunction(namedlambdavariable(), namedlambdavariable()))' due to data type mismatch: argument 3 requires int type, however, 'lambdafunction((CAST(namedlambdavariable() AS BIGINT) + namedlambdavariable()), namedlambdavariable(), namedlambdavariable())' is of bigint type.; line 3 pos 0;

如何使它正常工作？

Answer 1

您需要在累加器中强制转换值，例如浮动：

a = sc.parallelize([("a", [1,1,1]),
                    ("a", [2,2])])
a = a.toDF(["g", "arr_val"])

a.registerTempTable('a')

sql = """
select 
aggregate(arr_val, cast(0 as float), (acc, x) -> acc + cast(x as float)) as sum
from a
"""

spark.sql(sql).show()

使用聚合函数对列内的数组求和

1 个答案: