Question

我有两个表，p_to_v映射，g_to_v映射。

scala> val p_to_v = Seq(("p1", "v1"), ("p1", "v2"), ("p2", "v1")).toDF("p", "v")
scala> p_to_v.show
+---+---+
|  p|  v|
+---+---+
| p1| v1|
| p1| v2|
| p2| v1|
+---+---+

“ p1”映射到[v1，v2]

“ p2”映射到[v1]

scala> val g_to_v = Seq(("g1", "v1"), ("g2", "v1"), ("g2", "v2"), ("g3", "v2")).toDF("g", "v")
scala> g_to_v.show
+---+---+
|  g|  v|
+---+---+
| g1| v1|
| g2| v1|
| g2| v2|
| g3| v2|
+---+---+

“ g1”映射到[v1]

“ g2”映射到[v1，v2]

“ g3”映射到[v2]

我想获得所有 p 和 g 组合，其中p的对应 v 映射是 v g

的映射
p1 [v1, v2] - g2 [v1, v2] p2 [v1] - g1 [v1] p2 [v1] - g2 [v1, v2]

我怎么能得到？

Answer 1

这非常简单。您需要使用groupBy和简单的内部联接

scala> val p_to_v = Seq(("p1", "v1"), ("p1", "v2"), ("p2", "v1")).toDF("p", "v")
19/10/16 22:11:55 WARN metastore: Failed to connect to the MetaStore Server...
p_to_v: org.apache.spark.sql.DataFrame = [p: string, v: string]

scala> val g_to_v = Seq(("g1", "v1"), ("g2", "v1"), ("g2", "v2"), ("g3", "v2")).toDF("g", "v")
g_to_v: org.apache.spark.sql.DataFrame = [g: string, v: string]

现在通过分组进行分组

scala> val pv = p_to_v.groupBy($"p").agg(collect_list("v").as("pv"))
pv: org.apache.spark.sql.DataFrame = pv = [p: string, pv: array<string>]

scala> val gv = g_to_v.groupBy($"g").agg(collect_list("v").as("gv"))
gv: org.apache.spark.sql.DataFrame = [g: string, gv: array<string>]

scala> pv.show
+---+--------+
|  p|      pv|
+---+--------+
| p2|    [v1]|
| p1|[v1, v2]|
+---+--------+

scala> gv.show
+---+--------+
|  g|      gv|
+---+--------+
| g2|[v2, v1]|
| g3|    [v2]|
| g1|    [v1]|
+---+--------+

创建用于查找子集的UDF

import org.apache.spark.sql.Row
import org.apache.spark.sql.functions._

def subLis(ar1: Seq[Row], ar2: Seq[Row]) = ar1.toSet.subsetOf(ar2.toSet)
    subLis: (ar1: Seq[org.apache.spark.sql.Row], ar2: Seq[org.apache.spark.sql.Row])Boolean

val subLisUDF = udf(subLis _)
    UserDefinedFunction(<function2>,BooleanType,None)

现在您可以执行交叉连接并应用UDF

spark.conf.set("spark.sql.crossJoin.enabled", "true")
pv.join(gv).withColumn("newdsa", subLisUDF($"pv", $"gv")).filter($"newdsa").show

+---+--------+---+--------+------+
|  p|      pv|  g|      gv|newdsa|
+---+--------+---+--------+------+
| p2|    [v1]| g2|[v2, v1]|  true|
| p1|[v1, v2]| g2|[v2, v1]|  true|
| p2|    [v1]| g1|    [v1]|  true|
+---+--------+---+--------+------+

或有条件加入

pv.join(gv, pv("pv") === gv("gv") || subLisUDF($"pv", $"gv")).show

+---+--------+---+--------+
|  p|      pv|  g|      gv|
+---+--------+---+--------+
| p2|    [v1]| g2|[v1, v2]|
| p1|[v1, v2]| g2|[v1, v2]|
| p2|    [v1]| g1|    [v1]|
+---+--------+---+--------+

尝试两种方法，并选择性能最好的一种。

Spark：如何根据子集条件过滤数据

1 个答案: