将空值替换为pyspark中特定列的平均值

Question

我想将null值替换为age和height列的均值。我知道有一个帖子 Fill Pyspark dataframe column null values with average value from same column 但是在这篇文章中，给定的函数抛出错误。

df = spark.createDataFrame([(1, 'John', 1.79, 28,'M', 'Doctor'),
                    (2, 'Steve', 1.78, 45,'M', None),
                    (3, 'Emma', 1.75, None, None, None),
                    (4, 'Ashley',1.6, 33,'F', 'Analyst'),
                    (5, 'Olivia', 1.8, 54,'F', 'Teacher'),
                    (6, 'Hannah', 1.82, None, 'F', None),
                    (7, 'William', 1.7, 42,'M', 'Engineer'),
                    (None,None,None,None,None,None),
                    (8,'Ethan',1.55,38,'M','Doctor'),
                    (9,'Hannah',1.65,None,'F','Doctor')]
                   , ['Id', 'Name', 'Height', 'Age', 'Gender', 'Profession'])

给定帖子中的功能

def fill_with_mean(df, exclude=set()): 
stats = df.agg(*(
    avg(c).alias(c) for c in df.columns if c not in exclude
))
return df.na.fill(stats.first().asDict())
fill_with_mean(df, ["Age", "Height"])

当我运行此功能时，它会显示

NameError：未定义名称“ avg”

有人可以解决此问题吗？谢谢。

Answer 1

固定示例。它以您期望的方式对我有效！

from pyspark.sql.functions import avg

df = spark.createDataFrame(
    [
        (1, 'John', 1.79, 28, 'M', 'Doctor'),
        (2, 'Steve', 1.78, 45, 'M', None),
        (3, 'Emma', 1.75, None, None, None),
        (4, 'Ashley', 1.6, 33, 'F', 'Analyst'),
        (5, 'Olivia', 1.8, 54, 'F', 'Teacher'),
        (6, 'Hannah', 1.82, None, 'F', None),
        (7, 'William', 1.7, 42, 'M', 'Engineer'),
        (None, None, None, None, None, None),
        (8, 'Ethan', 1.55, 38, 'M', 'Doctor'),
        (9, 'Hannah', 1.65, None, 'F', 'Doctor')
    ],
    ['Id', 'Name', 'Height', 'Age', 'Gender', 'Profession']
)


def fill_with_mean(this_df, exclude=set()):

    stats = this_df.agg(*(avg(c).alias(c) for c in this_df.columns if c not in exclude))
    return this_df.na.fill(stats.first().asDict())



res = fill_with_mean(df, ["Gender", "Profession", "Id", "Name"])
res.show()