Question

我正在尝试使用 pyspark 在下面阅读的 JSON

test.json
{
  "Transactions": [
    {
      "ST": {
        "ST01": { "type": "271"},
        "ST02": {"type": "1001"},
        "ST03": {"type": "005010X279A1"}
      }
    }
  ]
}
+++++++++++++++++++++++++++++++++++
from pyspark.sql.types import *
from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = SparkSession.builder.appName("Spark - JSON read").master("local[*]") \
    .config("spark.driver.bindAddress", "localhost") \
    .getOrCreate()

ST = StructType([
        StructField("ST01", StructType([StructField("type", StringType())])),
        StructField("ST02", StructType([StructField("type", StringType())])),
        StructField("ST03", StructType([StructField("type", StringType())])),
])
ST1 = StructType([
        StructField("ST01", StringType()),
        StructField("ST02", StringType()),
        StructField("ST03", StringType()),
])

Json_schema = StructType()
Json_schema.add("ST", ST1)
# Json_schema.add("ST", ST)
Schema = StructType([StructField("Transactions", ArrayType(Json_schema))])
df1 = spark.read.option("multiline", "true").json("test.json", schema = Schema)
df1.select(F.explode("Transactions")).select("col.*").select("ST.*").show(truncate=False)

我想要的输出就像下面的类型值必须是列值

+-----+------+------------+
|ST01 |ST02  |ST03        |
+-----+------+------------+
|271  |1001  |005010X279A1|
+------------+------------+

但使用 ST 或 ST1 架构

With ST --> each column is a struct field
+-----+------+--------------+
|ST01 |ST02  |ST03          |
+-----+------+--------------+
|[271]|[1001]|[005010X279A1]|
+-----+------+--------------+

With ST1 --> its a JSON value for ST01, ST02 and ST03 cols
+--------------+---------------+-----------------------+
|ST01          |ST02           |ST03                   |
+--------------+---------------+-----------------------+
|{"type":"271"}|{"type":"1001"}|{"type":"005010X279A1"}|
+--------------+---------------+-----------------------+

我可以做 ST01.* 并为其取别名，但我作为输入获得的 JSON 是动态的，它可能包含也可能不包含所有三个标签。

有什么想法吗？

Answer 1

由于您的 JSON js 是动态的并且可能不包含所有三个标签，因此一种“动态”方法是对现有列使用 for 循环。一旦有了列名，你就可以extract JSON object或using expression，像这样

df2 = df1.select(F.explode("Transactions")).select("col.*").select("ST.*")

# with ST schema (struct type)
for col in df2.columns:
  df2 = df2.withColumn(col, F.expr(f'{col}.type'))

# with ST1 schema (JSON string type)
for col in df2.columns:
  df2 = df2.withColumn(col, F.get_json_object(col, '$.type'))

结果：

+----+----+------------+
|ST01|ST02|ST03        |
+----+----+------------+
|271 |1001|005010X279A1|
+----+----+------------+

Pyspark - 读取 json 文件并返回数据帧

1 个答案: