修改

Question

如何规范化主要由嵌套数组组成的火花数据帧？

case class FooBar(id:String, foo:Seq[String], bar:String, baz: Seq[String])
val f = Seq(FooBar("thinga", Seq("1 "), "1 2 3 ", Seq("2 ")),
    FooBar("thinga", Seq("1 2 3 4 "), " 0 0 0 ", Seq("2 3 4 5 ")),
    FooBar("thingb", Seq("1 2 "), "1 2 3 4 5 ", Seq("1 2 ")),
    FooBar("thingb", Seq("0 ", "0 ", "0 "), "1 2 3 4 5 ", Seq("1 2 3 "))).toDS
f.printSchema
f.show(false)
+------+------------+----------+----------+
|    id|         foo|       bar|       baz|
+------+------------+----------+----------+
|thinga|        [1 ]|    1 2 3 |      [2 ]|
|thinga|  [1 2 3 4 ]|    0 0 0 |[2 3 4 5 ]|
|thingb|      [1 2 ]|1 2 3 4 5 |    [1 2 ]|
|thingb|[0 , 0 , 0 ]|1 2 3 4 5 |  [1 2 3 ]|
+------+------------+----------+----------+


scala> f.printSchema
root
 |-- id: string (nullable = true)
 |-- foo: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- bar: string (nullable = true)
 |-- baz: array (nullable = true)
 |    |-- element: string (containsNull = true)

我想要像explode那样保留（id，foo，bar，baz）的模式，但是为数组的每个值返回一条单独的记录。最终结果不应再包含数组。

Foo和baz有关系。他们的顺序不得扭曲。它们总是具有相同的长度，foo的第一个值与baz的第一个值相关 - 依此类推。也许我应该先将它们组合成一个列/结构？

最终结果应该类似于：

+------+------------+----------+----------+
|    id|         foo|       bar|       baz|
+------+------------+----------+----------+
|thinga|        1   |    1     |      2   |
|thinga|        1   |      2   |      2   |
|thinga|        1   |        3 |      2   |    
|thinga|  1         |    0     |2         |
|thinga|  2         |    0     |3         |
|thinga|  3         |    0     |4         |
|thinga|  4         |    0     |5         |
|thinga|  1         |    0     |2         |
|thinga|  2         |    0     |3         |
|thinga|  3         |    0     |4         |
|thinga|  4         |    0     |5         |

|thinga|  1         |    0     |2         |
|thinga|  2         |    0     |3         |
|thinga|  3         |    0     |4         |
|thinga|  4         |    0     |5         |

....

|thingb|0           |1         |       1  |
|thingb|0           |2         |       2  |
|thingb|0           |3         |       3  |
|thingb|0           |4         |       1  |
|thingb|0           |5         |       2  |
|thingb|0           |1         |       3  |
|thingb|0           |2         |       1  |
|thingb|0           |3         |       2  |
|thingb|0           |4         |       3  |
|thingb|0           |5         |       1  |
|thingb|0           |1         |       2  |
|thingb|0           |2         |       3  |
|thingb|0           |3         |       1  |
|thingb|0           |4         |       2  |
|thingb|0           |5         |       3  |
+------+------------+----------+----------+

修改

部分相关问题 - Explode (transpose?) multiple columns in Spark SQL table

Answer 1

根据我们的讨论（请查看初始帖子下的评论），以下数据应该有效：

+------+-------+---------+-------+
|    id|    foo|      bar|    baz|
+------+-------+---------+-------+
|thinga|      1|    1 2 3|      2|
|thinga|1 2 3 4|    0 0 0|2 3 4 5|
|thingb|    1 2|1 2 3 4 5|    1 2|
|thingb|  0 0 0|1 2 3 4 5|  1 2 3|
+------+-------+---------+-------+

然后初始化代码如下：

case class FooBar(id:String, foo:String, bar:String, baz: String)
val f = Seq(FooBar("thinga", "1", "1 2 3", "2"),
  FooBar("thinga", "1 2 3 4", "0 0 0", "2 3 4 5"),
  FooBar("thingb", "1 2", "1 2 3 4 5", "1 2"),
  FooBar("thingb", "0 0 0", "1 2 3 4 5", "1 2 3")).toDS()

如果是这种情况，则此代码应产生理想的结果：

f
  .withColumn("foo", split($"foo", " "))
  .withColumn("baz", split($"baz", " "))
  .withColumn("bar", explode(split($"bar", " ")))
  .map { case Row(id: String, foo: Seq[String], bar: String, baz: Seq[String]) =>
    val c = for ((f, b) <- foo.zip(baz)) yield {
      (f, b)
    }
    (id, bar, c)
  }.toDF(cols: _*)
  .withColumn("foo+baz", explode($"foo+baz"))
  .withColumn("foo", $"foo+baz._1")
  .withColumn("baz", $"foo+baz._2")
  .drop($"foo+bar")
  .select("id", "foo", "bar", "baz")
  .show(100)

前两次转换将拆分空格分隔的列 foo 和 baz 。由于列 bar 是字符串，我们需要转换为带有split的数组，然后将其展开。 Map将返回（id，bar，c）元组，其中c是元组序列（foo，bar）。在地图之后我们得到下一个输出：

+------+---+--------------------+
|    id|bar|             foo+baz|
+------+---+--------------------+
|thinga|  1|             [[1,2]]|
|thinga|  2|             [[1,2]]|
|thinga|  3|             [[1,2]]|
|thinga|  0|[[1,2], [2,3], [3...|
|thinga|  0|[[1,2], [2,3], [3...|
|thinga|  0|[[1,2], [2,3], [3...|
|thingb|  1|      [[1,1], [2,2]]|
|thingb|  2|      [[1,1], [2,2]]|
|thingb|  3|      [[1,1], [2,2]]|
|thingb|  4|      [[1,1], [2,2]]|
|thingb|  5|      [[1,1], [2,2]]|
|thingb|  1|[[0,1], [0,2], [0...|
|thingb|  2|[[0,1], [0,2], [0...|
|thingb|  3|[[0,1], [0,2], [0...|
|thingb|  4|[[0,1], [0,2], [0...|
|thingb|  5|[[0,1], [0,2], [0...|
+------+---+--------------------+

接下来我们再次使用“foo + baz”进行爆炸，以提取最终元组。现在输出看起来如下：

+------+---+-------+
|    id|bar|foo+baz|
+------+---+-------+
|thinga|  1|  [1,2]|
|thinga|  2|  [1,2]|
|thinga|  3|  [1,2]|
|thinga|  0|  [1,2]|
|thinga|  0|  [2,3]|
|thinga|  0|  [3,4]|
|thinga|  0|  [4,5]|
|thinga|  0|  [1,2]|
.....
|thingb|  1|  [0,2]|
|thingb|  1|  [0,3]|
|thingb|  2|  [0,1]|
|thingb|  2|  [0,2]|
|thingb|  2|  [0,3]|
|thingb|  3|  [0,1]|
|thingb|  3|  [0,2]|
|thingb|  3|  [0,3]|
|thingb|  4|  [0,1]|
|thingb|  4|  [0,2]|
|thingb|  4|  [0,3]|
|thingb|  5|  [0,1]|
|thingb|  5|  [0,2]|
|thingb|  5|  [0,3]|
+------+---+-------+

最后，我们填写 foo + baz._1 和 foo + baz._2 foo 和 baz 列>分别。这将是最终输出：

+------+---+---+---+
|    id|foo|bar|baz|
+------+---+---+---+
|thinga|  1|  1|  2|
|thinga|  1|  2|  2|
|thinga|  1|  3|  2|
|thinga|  1|  0|  2|
|thinga|  2|  0|  3|
|thinga|  3|  0|  4|
|thinga|  4|  0|  5|
|thinga|  1|  0|  2|
|thinga|  2|  0|  3|
|thinga|  3|  0|  4|
|thinga|  4|  0|  5|
|thinga|  1|  0|  2|
|thinga|  2|  0|  3|
|thinga|  3|  0|  4|
|thinga|  4|  0|  5|
|thingb|  1|  1|  1|
|thingb|  2|  1|  2|
|thingb|  1|  2|  1|
|thingb|  2|  2|  2|
|thingb|  1|  3|  1|
|thingb|  2|  3|  2|
|thingb|  1|  4|  1|
|thingb|  2|  4|  2|
|thingb|  1|  5|  1|
|thingb|  2|  5|  2|
|thingb|  0|  1|  1|
|thingb|  0|  1|  2|
|thingb|  0|  1|  3|
|thingb|  0|  2|  1|
|thingb|  0|  2|  2|
|thingb|  0|  2|  3|
|thingb|  0|  3|  1|
|thingb|  0|  3|  2|
|thingb|  0|  3|  3|
|thingb|  0|  4|  1|
|thingb|  0|  4|  2|
|thingb|  0|  4|  3|
|thingb|  0|  5|  1|
|thingb|  0|  5|  2|
|thingb|  0|  5|  3|
+------+---+---+---+

spark规范化数组的数据帧

修改

1 个答案: