Question

我正在pyspark中读取json文件，该文件的第三层嵌套的结构名称对于每一行都是不同的。

Schema looks something like below

 |-- A: string (nullable = true)
 |-- Plugins: struct (nullable = true)
 |    |-- RfS: struct (nullable = true)
 |    |    |-- A: string (nullable = true)
 |    |    |-- B: string (nullable = true)
 |    |-- RtW: struct (nullable = true)
 |    |    |-- A: string (nullable = true)
 |    |    |-- B: string (nullable = true)

which I want to convert to dataframe of following schma

 |-- A: string (nullable = true)
 |-- Plugins: struct (nullable = true)
 |-- A: string (nullable = true)
 |-- B: string (nullable = true)

Plugins will contain value from struct name RfS/RtW etc.

我读取了数据并删除了第一层嵌套

jsonData = """{
  "A" : "some A",
  "Plugins": {
    "RfS": {
      "A" : "RfSA",
      "B" : "RfSB"
    },
    "RtW" : {
      "A" : "RtWA",
      "B" : "RtWA"
    }
  }
}"""

df = spark.read.json(sc.parallelize([jsonData]))
no_plug_cols = ["A"] # cols not in Plugins i.e A
plug_df = df.select("A", "Plugins.*")

# plug_df.printSchema()
# root
#  |-- A: string (nullable = true)
#  |-- RfS: struct (nullable = true)
#  |    |-- A: string (nullable = true)
#  |    |-- B: string (nullable = true)
#  |-- RtW: struct (nullable = true)
#  |    |-- A: string (nullable = true)
#  |    |-- B: string (nullable = true)

遵循这里的答案之一，我认为获得以下内容很简单

icols = [(col(f"{c}.A").alias(f"{c}.A"), col(f"{c}.B").alias(f"{c}.B")) for c in (set(plug_df.columns) - set(no_plug_cols))]

# we use chain to flatten icols which is a list of tuples   
plug_df.select(no_plug_cols + list(chain(*icols))).show()

# +------+-----+-----+-----+-----+
# |     A|RfS.A|RfS.B|RtW.A|RtW.B|
# +------+-----+-----+-----+-----+
# |some A| RfSA| RfSB| RtWA| RtWA|
# +------+-----+-----+-----+-----+

有没有办法代替上面的输出，我可以将RfS / RtW作为具有所需名称的列值，以便输出看起来像下面的样子。仅当我在转换为上述格式后使用数据透视表转换数据

时，

# +------+-----  +-----+-----+
# |     A|Plugins|    A|    B|
# +------+-------+-----+-----+
# |some A| RfS   | RfSA| RfSB|
# +------+-------+-----+-----+
# |some A| RtW   | RfWA| RtWA|
# +------+-------+-----+-----+

在pyspark中爆炸嵌套的Struct，并使用结构名称作为列

0 个答案: