Question

我有以下两个数据框，我想基于col A加入该数据框

df1：

+------+--------+-------+
|  A   |   B    |   C   |
+------+--------+-------+
| a1   |   5    |   asd |
| a2   |   12   |   asd |
+------+--------+-------+

df2：

+------+--------+-------+
|  A   |   B    |   D   |
+------+--------+-------+
|  a1  |   8    |   qwe |
|  a2  |   10   |   qwe |
+------+--------+-------+

由于B列相同，因此假设在两者之间进行选择是一种逻辑，例如选择

+------+--------+------+-----+
|  A   |   B    |  C   |  D  |
+------+--------+------+-----+
|  a1  |   8    |  asd | qwe |
|  a2  |   12   |  asd | qwe |
+------+--------+-------+----+

实现此目标的简单方法是：

val _df1 = df1.withColumnRenamed("B","B_df1")
val _df2 = df2.withColumnRenamed("B", "B_df2)
_df1.join(_df2, Seq("A"))
    .withColumn("B", when(col("B_df1") > col("B_df2"), 
 col("B_df1"))
    .otherwise(col("B_df2"))
    .drop(col("B_df1")
    .drop("B_df2")

有没有更好的方法来实现此目的而无需重命名和删除列？

Answer 1

这是使用selectExpr的另一种方法。它节省了删除列的任何工作。

import spark.implicits._

val df1 = Seq(("a1",5,"asd"),
              ("a2",12,"asd")
              ).toDF("A","B","C")

val df2 = Seq(("a1",8,"qwe"),
              ("a2",10,"qwe")
              ).toDF("A","B","D")


import org.apache.spark.sql.functions.col

df1.as("a").join(df2.as("b"), col("a.A") === col("b.A")).selectExpr("a.A AS A",
               "CASE WHEN a.B>b.B THEN a.B ELSE b.B END AS B",
               "a.C AS C",
               "b.D AS D").show()

Spark在非联接列上具有条件联接

1 个答案: