Question

我的重复行可能在PySpark数据框中包含相同的数据或缺少值。我编写的代码非常慢，不能作为分布式系统使用。有谁知道如何从PySpark数据框的重复行中保留单个唯一值，该数据可以作为分布式系统运行并具有快速的处理时间？

我已经编写了完整的Pyspark代码，并且此代码可以正常工作。但是处理时间确实很慢，无法在Spark群集上使用它。

'''

# Columns of duplicate Rows of DF

dup_columns = df.columns

for row_value in df_duplicates.rdd.toLocalIterator():
    print(row_value)

# Match duplicates using std name and create RDD

    fill_duplicated_rdd = ((df.where((sf.col("stdname") == row_value['stdname'] ))
                        .where(sf.col("stdaddress")== row_value['stdaddress']))
                   .rdd.map(fill_duplicates))

    # Creating feature names for the same RDD 

    fill_duplicated_rdd_col_names = (((df.where((sf.col("stdname") == row_value['stdname']) &
                                    (sf.col("stdaddress")== row_value['stdaddress'])))
                   .rdd.map(fill_duplicated_columns_extract)).first())

    # Creating DF using the previous RDD
    # This DF stores value of a single set of matching duplicate rows

    df_streamline = fill_duplicated_rdd.toDF(fill_duplicated_rdd_col_names)

    for column in df_streamline.columns:
        try:

            col_value = ([str(value[column]) for value in 
                      df_streamline.select(col(column)).distinct().rdd.toLocalIterator() if value[column] != ""])        

            if len(col_value) >= 1:
                # non null or empty value of a column store here
                # This value is a no duplicate distinct value
                col_value = col_value[0]
                #print(col_value)

                # The non-duplicate distinct value of the column is stored back to 
                # replace any rows in the PySpark DF that were empty.

                df_dedup = (df_dedup
              .withColumn(column,sf.when((sf.col("stdname") == row_value['stdname'])
                                      & (sf.col("stdaddress")== row_value['stdaddress'])
                                         ,col_value)
                          .otherwise(df_dedup[column])))

            #print(col_value)
        except:
            print("None")

'''

没有错误消息，但是代码运行非常缓慢。我想要一个在PySpark DF中用唯一值填充行的解决方案。它可以用值的偶数模式填充行

Answer 1

“”“

shadowPath

“”“

从PySpark数据框中的重复行中提取和替换值

1 个答案: