Pyspark-如何使用RDD进行分组和聚合

Question

我有数据框

df_1 = spark.createDataFrame(
[(1, [(1), (4), (2) ,(3)])],
("id", "transactions")
)

df_2 = spark.createDataFrame(
[([ (1),(2),(3)], 2.0) ],
("items", "cost")
)

df_1_rdd= df_1.rdd
df_2_rdd=df_2.rdd

我有一个UDF来检查数组的所有元素是否在另一个数组中

@udf("boolean")
def contains_all(x, y):
    if x is not None and y is not None:
        return set(y).issubset(set(x))

如何使用rdd分组和汇总？

costs=(df_1_rdd
.cartesian(df_2_rdd).groupBy("id", "transactions")
.agg(sum_(when(
   contains_all("transactions", "items"), col("support")
 )).alias("score")))