在spark数据框中，我有1列，其中包含作为行的列表列表。我想将字符串列表合并为一个。
INPUT DATAFRAME:
+-------+--------------------+
| name |friends |
+-------+--------------------+
| Jim |[["C","A"]["B","C"]]|
+-------+--------------------+
| Bill |[["E","A"]["F","L"]]|
+-------+--------------------+
| Kim |[["C","K"]["L","G"]]|
+-------+--------------------+
OUTPUT DATAFRAME:
+-------+--------------------+
| name |friends |
+-------+--------------------+
| Jim |["C","A","B"] |
+-------+--------------------+
| Bill |["E","A","F","L"] |
+-------+--------------------+
| Kim |["C","K","L","G"] |
+-------+--------------------+
我想将列表列表合并为单个列表，并删除重复列表。 预先感谢
答案 0 :(得分：2)
我认为您可以依靠explode来解构列表，并依靠collect_set来重建它们：
import pyspark
from pyspark.sql import SparkSession
from pyspark import SparkContext
import pandas as pd
from pyspark.sql import functions as F
from pyspark.sql import Window
sc = SparkContext.getOrCreate()
spark = SparkSession(sc)
columns = ['name', 'friends']
data = [("Jim", [["C","A"], ["B","C"]]), ("Bill", [["E","A"], ["F","L"]]), ("Kim", [["C","K"], ["L","G"]])]
pd_data = pd.DataFrame.from_records(data=data, columns=columns)
spark_data = spark.createDataFrame(pd_data)
first_explode = spark_data.withColumn("first_explode", F.explode((F.col("friends"))))
first_explode.show()
+----+----------------+-------------+
|name| friends|first_explode|
+----+----------------+-------------+
| Jim|[[C, A], [B, C]]| [C, A]|
| Jim|[[C, A], [B, C]]| [B, C]|
|Bill|[[E, A], [F, L]]| [E, A]|
|Bill|[[E, A], [F, L]]| [F, L]|
| Kim|[[C, K], [L, G]]| [C, K]|
| Kim|[[C, K], [L, G]]| [L, G]|
+----+----------------+-------------+
一级解构。现在是第二个：
second_explode = first_explode.withColumn("second_explode", F.explode(F.col("first_explode")))
second_explode.show()
+----+----------------+-------------+--------------+
|name| friends|first_explode|second_explode|
+----+----------------+-------------+--------------+
| Jim|[[C, A], [B, C]]| [C, A]| C|
| Jim|[[C, A], [B, C]]| [C, A]| A|
| Jim|[[C, A], [B, C]]| [B, C]| B|
| Jim|[[C, A], [B, C]]| [B, C]| C|
|Bill|[[E, A], [F, L]]| [E, A]| E|
|Bill|[[E, A], [F, L]]| [E, A]| A|
|Bill|[[E, A], [F, L]]| [F, L]| F|
|Bill|[[E, A], [F, L]]| [F, L]| L|
| Kim|[[C, K], [L, G]]| [C, K]| C|
| Kim|[[C, K], [L, G]]| [C, K]| K|
| Kim|[[C, K], [L, G]]| [L, G]| L|
| Kim|[[C, K], [L, G]]| [L, G]| G|
+----+----------------+-------------+--------------+
重建列表，丢弃重复项：
grouped = second_explode.groupBy("name").agg(F.collect_set(F.col("second_explode")).alias("friends"))
grouped.show()
+----+------------+
|name| friends|
+----+------------+
| Jim| [C, B, A]|
|Bill|[F, E, A, L]|
| Kim|[K, C, G, L]|
+----+------------+