Question

我正在尝试解析JSON数据，因为我为此编写了自定义架构。通过添加架构解析数据或不添加架构我得到以下错误：

{
    "queryResults": {
        "searchResponse": {
            "response": {
                "docs": [{
                    "transactions": [{
                        "recordDate": "2010-02-02 00:00:00",
                        "code": "PGM/",
                        "description": "Recordation of Patent Grant Mailed"
                    }, {
                        "recordDate": "2010-01-13 00:00:00",
                        "code": "WPIR",
                        "description": "Issue Notification Mailed"
                    }, {
                        "recordDate": "2009-12-17 00:00:00",
                        "code": "R1021",
                        "description": "Receipt into Pubs"
                    }]
                }]
            }
        }
    }
}

以下是我的示例数据：

val schema=StructType(List(
      StructField("queryResults",StructType(
        List(StructField("searchResponse",StructType(
          List(StructField("response",StructType(
            List(StructField("docs",ArrayType(StructType(
              List(
                StructField("appCustNumber", StringType, nullable = true),
                StructField("transactions",ArrayType(StructType(
                  List
                  (
                    StructField("code", StringType, nullable = true),
                    StructField("description", StringType, nullable = true),
                    StructField("recordDate", StringType, nullable = true)
                  )
                )))
              )
            ))))
          )))
        )))
      ))
    ))

这是我的架构：

val dff = sqlCotext.read.schema(schema).json("file locatiuon")
dff.select("queryResults.searchResponse.response.docs.transactions.code").show()

以下是我尝试获取数据的方式：

postgres

先谢谢。

Answer 1

试试这个......

val dfContent = dff.select(explode(dff("queryResults.searchResponse.response.docs.transactions"))).toDF("transaction")

val code = dfContent.select("transaction.code")

code.show(false)

由于数据类型不匹配导致：参数2在解析Json数据Spark SQL时需要整数类型错误

1 个答案: