Question

我正在尝试将一个数据帧插入另一个数据帧。

scala> addressOrigRenamed.show
+--------------+----------------------+-----------+-----------+
|orig_contactid|orig_contactaddresskey|orig_valueA|orig_valueB|
+--------------+----------------------+-----------+-----------+
|             1|                     1|         54|          3|
|             1|                     2|         55|          7|
+--------------+----------------------+-----------+-----------+
scala> dfNew.show
+---------+-----------------+------+------+
|contactId|contactaddresskey|valueA|valueB|
+---------+-----------------+------+------+
|        1|                2|    10|     9|
+---------+-----------------+------+------+
scala> val endDF = addressOrigRenamed.join(dfNew, $"orig_contactid" === $"contactid" && $"orig_contactaddresskey" === "$contactaddresskey", "fullouter").select(coalesce($"contactid", $"orig_contactid").alias("contactid"), coalesce($"contactaddresskey", $"orig_contactaddresskey").alias("contactaddresskey"), coalesce($"valueA", $"orig_valueA").alias("valueA"), coalesce($"valueB", $"orig_valueB").alias("valueB"))
scala> endDF.show
+---------+-----------------+------+------+
|contactid|contactaddresskey|valueA|valueB|
+---------+-----------------+------+------+
|        1|                1|    54|     3|
|        1|                2|    10|     9|
+---------+-----------------+------+------+

如您所见，这可行。但是语法是可怕的。这只是一个测试，我需要合并15-20个色谱柱。写coalesce(....).alias(...) 15-20确实是一个糟糕的选择。我该怎么写呢？

Answer 1

可以创建一系列合并函数：

scala> val joinedDF = addressOrigRenamed.join(dfNew, $"orig_contactid" === $"contactid" && $"orig_contactaddresskey" === "$contactaddresskey", "fullouter")
scala> val arr = dfNew.columns.map(x => {
         val y = "orig_" + x
         coalesce(joinedDF.col(x), joinedDF.col(y)).alias(x)
      })

然后您可以选择使用此arr，并记住要传播arr的元素：

scala> joinedDF.select(arr:_*).show 
+---------+-----------------+------+------+
|contactId|contactaddresskey|valueA|valueB|
+---------+-----------------+------+------+
|        1|                1|    54|     3|
|        1|                2|    10|     9|
+---------+-----------------+------+------+

Spark 2一次合并多列

1 个答案: