Question

我正在尝试将数据框转换为嵌套的json。

基本上最终的输出在字段“ id”级别，其他字段的嵌套json格式。

Json格式，使用字段“ rank”作为键，“ desc”和“ percent”作为值。感谢您的帮助！

源数据：-

val df = Seq(
  ("1", "ABC", "1","91.68"),
  ("1", "BCD", "2","89.03"),
  ("1", "DEF", "3","78.32"),
  ("1", "XYZ", "4","70.64")
).toDF("id", "desc", "rank", "percent")

+---+----+----+-------+
|id |desc|rank|percent|
+---+----+----+-------+
|1  |ABC |1   |91.68  |
|1  |BCD |2   |89.03  |
|1  |DEF |3   |78.32  |
|1  |XYZ |4   |70.64  |
+---+----+----+-------+

预期输出：

+---+--------------------------------------------------------------------------------------------------------------------------------------------------------+
|id |json                                                                                                                                                    |
+---+--------------------------------------------------------------------------------------------------------------------------------------------------------+
|1  |{"1":{"desc":"ABC","percent":"91.68"},"2":{"desc":"BCD","percent":"89.03"},"3":{"desc":"DEF","percent":"78.32"},"4":{"desc":"XYZ","percent":"70.64"}}   |
+---+--------------------------------------------------------------------------------------------------------------------------------------------------------+

Answer 1

您可以使用简单的UDF来根据需要修复json：

val toJsonUDF = udf((key: String, jsonString: String) =>  "{ " + key + ":" + jsonString + "}")

然后使用先前的UDF和withColumn方法使用to_json方法创建一个json列，该方法从给定的列中创建一个jsonString：

df.withColumn("json", toJsonUDF($"rank", to_json(struct("desc","percent"))))
  .select("id", "json")
  .groupBy("id")
  .agg(collect_list("json").as("json"))

输出

      +---+------------------------------------------------------------------------------------------------------------------------------------------------------------+
      |id |collect_list(json)                                                                                                                                          |
      +---+------------------------------------------------------------------------------------------------------------------------------------------------------------+
      |1  |[{ 1:{"desc":"ABC","percent":"91.68"}}, { 2:{"desc":"BCD","percent":"89.03"}}, { 3:{"desc":"DEF","percent":"78.32"}}, { 4:{"desc":"XYZ","percent":"70.64"}}]|
      +---+------------------------------------------------------------------------------------------------------------------------------------------------------------+

Spark Scala数据框列到嵌套的json

1 个答案: