我有2个数据框，其中包含以下条目： df1-
+----------+--------------------+
| id1| joinKey1|
+----------+--------------------+
|b000jz4hqo|[clickart, 950, 0...|
|b0006zf55o|[ca, internationa...|
|b00004tkvy|[noah, ark, activ...|
|b000g80lqo|[peachtree, sage,...|
|b0006se5bq|[singing, coach, ...|
|b000ehpzv8|[emc, retrospect,...|
|b00021xhzw|[adobe, effects, ...|
|b000gzwjgc|[acad, upgrade, d...|
|b0000dbykm|[mia, math, adven...|
|b00029bqa2|[disney, 1st, 2nd...|
|b0007prnjo|[paper, art, gift...|
|b000aazr5i|[nfs, maestro, so...|
|b000bhl1r8|[microsoft, sql, ...|
|b000i82j80|[spy, sweeper, sp...|
|b00006hmwc|[domino, designer...|
|b000in6u62|[omnioutliner, pr...|
|b000083k56|[compaq, comp, se...|
|b00006hvvo|[upg, sgms, 1000,...|
|b0000ycfcw|[human, body, top...|
|b00066dd5m|[school, zone, pe...|
+----------+--------------------+
df2：
+--------------------+--------------------+
| id2| joinKey2|
+--------------------+--------------------+
|http://www.google...|[spanish, vocabul...|
|http://www.google...|[topics, presents...|
|http://www.google...|[sierrahome, hse,...|
|http://www.google...|[adobe, cs3, prod...|
|http://www.google...|[equisys, premium...|
|http://www.google...|[quicken, r, quic...|
|http://www.google...|[sea, scene, livi...|
|http://www.google...|[autodesk, 34006,...|
|http://www.google...|[apple, garageban...|
|http://www.google...|[first, bible, st...|
|http://www.google...|[apple, apple, ma...|
|http://www.google...|[adobe, systems, ...|
|http://www.google...|[microsoft, 392, ...|
|http://www.google...|[panda, software,...|
|http://www.google...|[learn2, training...|
|http://www.google...|[family, tree, ma...|
|http://www.google...|[data, protection...|
|http://www.google...|[pencil, pal, big...|
|http://www.google...|[sos, aggregation...|
|http://www.google...|[fogware, publish...|
+--------------------+--------------------+
数据帧1和数据帧2在joinkey1和joinkey2中具有列表。 列表中的元素有些相同。 我想以joinkey1和joinkey2至少有1个共同点的条件加入这两个数据帧。
例如
joinkey1有
['clickart', '950', '000', 'premier', 'image', 'pack', 'dvd', 'rom', 'broderbund']
和joinkey2有
['clickart', '950000', 'premier', 'image', 'pack', 'dvd', 'rom']
2个列表具有6个公共元素，因此联接的数据框应添加包含这6个公共元素的列。
我想知道将2个数据框与列表上的条件连接起来的条件，以及如何插入仅给出共同元素的列。 这必须使用pyspark（spark版本2.4.0+）来完成
答案 0 :(得分：1)
下面的方法是什么？
交叉联接这两个数据框，使用array_intersect函数添加一列，然后过滤相交的结果列的大小> 0的联接数据集。
例如：
df1 = spark.read # ... Read your first source
df2 = spark.read # ... Read your other source
from pyspark.sql import functions as fn
joined = df1.crossJoin(df2). \
withColumn("common_join_keys", fn.array_intersect(fn.col("joinkey1"), fn.col("joinkey2")))
result = joined.filter(fn.size(fn.col("common_join_keys")) > 0) # your condition
result.show(truncate=False)