我们有两个表,其中第一个表包含每次运行中每个任务的代码。第二个表包含每次运行中所有任务的代码。如何运行Spark sql根据分隔符拆分第二个表中的列,并在第一个表的in和IN语句中使用它
表格看起来像这样
表f1
+-------+-----+--+
| runid | tid | |
+-------+-----+--+
| 1a | cb4 | |
| 1a | hb5 | |
| 1a | hb6 | |
| 1b | gh6 | |
| 1b | gh7 | |
| 1c | kl9 | |
+-------+-----+--+
表f2
+-------+-------------+
| runid | tid |
+-------+-------------+
| 1a | cb4,hb5,hb6 |
| 1b | gh6,gh7,gh8 |
+-------+-------------+
我尝试了拆分,但是它似乎没有用,并且regexp_extract似乎没有帮助
select e.* from f1 e inner join
f2 a
on e.runid=a.runid where e.runid in ('1a',
'1b') and e.tid in (select split(a.tid, '[,]') from f2)
所需的输出为
+-------+-----+
| runid | tid |
+-------+-----+
| 1a | cb4 |
| 1a | hb5 |
| 1a | hb6 |
| 1b | gh6 |
| 1b | gh7 |
+-------+-----+
就火花SQL而言,我是菜鸟。任何帮助将不胜感激
答案 0 :(得分:0)
将lateral view
与explode
结合使用,可获取每行一个tid,然后将其用于join
。
with exploded_f2 as
(select runid,tid,expl_tid
from f2
lateral view explode(split(tid,',')) tbl as expl_tid
)
select f1.*
from f1
join exploded_f2 f2 on f1.tid = f2.expl_tid
答案 1 :(得分:0)
这是另一个版本:
val df3 = df2.flatMap {x => x.getString(1).split(",")
.map(y => (x.getString(0),y))}.toDF("runid","tid2")
df3.show()
+-----+----+
|runid|tid2|
+-----+----+
| 1a| cb4|
| 1a| hb5|
| 1a| hb6|
| 1b| gh6|
| 1b| gh7|
| 1b| gh8|
+-----+----+
然后加入df1和df3
df1.join(df3, "runid").select($"runid", $"tid").distinct.show(false)
+-----+---+
|runid|tid|
+-----+---+
|1a |hb5|
|1b |gh7|
|1b |gh6|
|1a |hb6|
|1a |cb4|
+-----+---+
答案 2 :(得分:0)
from pyspark.sql.functions import *
from pyspark.sql.types import *
schema=StructType([StructField("runid",StringType()),StructField("tid",StringType())])
f1=spark.read.format("csv").schema(schema).option("header","true").option("delimiter","|").load("c:/tmp/f1.csv")
f2=spark.read.format("csv").schema(schema).option("header","true").option("delimiter","|").load("c:/tmp/f2.csv")
f2_alter=(f2.withColumn("tid_explode",explode(split(f2.tid,"[,]")))).select("runid",col("tid_explode").alias("tid"))
df2=f1.join(f2_alter,["runid","tid"]).show()
+-----+---+
|runid|tid|
+-----+---+
| 1a|cb4|
| 1a|hb5|
| 1a|hb6|
| 1b|gh6|
| 1b|gh7|
+-----+---+