Question

我已经创建了如下数据框：

from pyspark.sql import Row
l = [('Ankit','25','Ankit','Ankit'),('Jalfaizy','2.2','Jalfaizy',"aa"),('saurabh','230','saurabh',"bb"),('Bala','26',"aa","bb")]
rdd = sc.parallelize(l)
people = rdd.map(lambda x: Row(name=x[0], ages=x[1],lname=x[2],mname=x[3]))
schemaPeople = sqlContext.createDataFrame(people)
schemaPeople.show()

+----+--------+-----+--------+
|ages|   lname|mname|    name|
+----+--------+-----+--------+
|  25|   Ankit|Ankit|   Ankit|
| 2.2|Jalfaizy|   aa|Jalfaizy|
| 230| saurabh|   bb| saurabh|
|  26|      aa|   bb|    Bala|
+----+--------+-----+--------+

我想查找所有社区的每列平均长度，即低于我的预期输出。即特定列中的字符总数/行数

+----+--------+-----+--------+
|ages|   lname|mname|    name|
+----+--------+-----+--------+
|2.5 | 5.5    | 2.75 |  6    |
+----+--------+-----+--------+

Answer 1

这实际上很简单。我们将为列length使用投影，并为avg使用聚合：

from pyspark.sql.functions import length, col, avg

selection = ['lname','mname','name']

schemaPeople \
    .select(*(length(col(c)).alias(c) for c in selection)) \
    .agg(*(avg(col(c)).alias(c) for c in selection)).show()

# +-----+-----+----+
# |lname|mname|name|
# +-----+-----+----+
# |  5.5| 2.75| 6.0|
# +-----+-----+----+

这样，您将能够动态传递列名。实际上，我们在这里所做的是打开参数列表（selection）

参考：Control Flow Tools - Unpacking Argument Lists。

Answer 2

我认为您可以为各个长度创建新行，然后对数据框进行分组。然后您将得到类似以下内容的

df_new = spark.createDataFrame([
( "25","Ankit","Ankit","Ankit"),( "2.2","Jalfaizy","aa","Jalfaizy"),
("230","saurabh","bb","saurabh") ,( "26","aa","bb","Bala")
], ("age", "lname","mname","name"))

df_new.withColumn("len_age",length(col("age"))).withColumn("len_lname",length(col("lname")))\
.withColumn("len_mname",length(col("mname"))).withColumn("len_name",length(col("name")))\
.groupBy().agg(avg("len_age"),avg("len_lname"),avg("len_mname"),avg("len_name")).show()

结果：

+------------+--------------+--------------+-------------+
|avg(len_age)|avg(len_lname)|avg(len_mname)|avg(len_name)|
+------------+--------------+--------------+-------------+
|         2.5|           5.5|          2.75|          6.0|
+------------+--------------+--------------+-------------+

Answer 3

猜测，在Scala中可以通过这种方式完成，作者可以将其转换为Python：

val averageColumnList = List("age", "lname", "mname", "name")
val columns = averageColumnList.map(name => avg(length(col(name))))
val result = df.select(columns: _*)

如何在pyspark中查找每列的平均长度？

3 个答案: