Question

我有一个数据框df2，例如：

ID | data
--------
1 | New
3 | New 
5 | New

和主数据帧df1：

ID | data | more
----------------
1 | OLD | a
2 | OLD | b
3 | OLD | c
4 | OLD | d
5 | OLD | e

我想实现某种目标：

ID | data | more
----------------
1 | NEW | a
2 | OLD | b
3 | NEW | c
4 | OLD | d
5 | NEW | e

我想基于df2更新df1，并保留df2中不存在的原始值。

是否有比使用isin更快的方法？当df1和df2都很大时，Isin非常慢。

Answer 1

具有左联接和“ coalesce”：

val df1 = Seq(
  (1, "OLD", "a"),
  (2, "OLD", "b"),
  (3, "OLD", "c"),
  (4, "OLD", "d"),
  (5, "OLD", "e")).toDF("ID", "data", "more")

val df2 = Seq(
  (1, "New"),
  (3, "New"),
  (5, "New")).toDF("ID", "data")

// action
val result = df1.alias("df1")
  .join(
    df2.alias("df2"),$"df2.ID" === $"df1.ID", "left")
  .select($"df1.ID",
    coalesce($"df2.data", $"df1.data").alias("data"),
    $"more")

输出：

+---+----+----+
|ID |data|more|
+---+----+----+
|1  |New |a   |
|2  |OLD |b   |
|3  |New |c   |
|4  |OLD |d   |
|5  |New |e   |
+---+----+----+

如何基于Spark中的另一个数据框修改数据框行？

1 个答案: