Question

我在S3存储桶中具有以下格式的JSON，我正尝试使用Athena从“字段”键中仅提取“ id”，“ label”和“ value”。我尝试了ARRAY-MAP，但没有成功。另外，在“值”字段上-我希望将内容捕获为简单文本，而忽略其中的任何列表/词典。

我也不想为这些JSON创建任何Hive模式，并尽可能地寻找Presto SQL解决方案。

{  
    "reports":{  
        "client":{  
            "pdf":"https://reports.s3-accelerate.amazonaws.com/looks/123/reports/client.pdf",
            "html":"https://api.com/looks/123/reports/client.html"
        },
        "public":{  
            "pdf":"https://s3.amazonaws.com/reports.com/looks/123/reports/public.pdf",
            "html":"https://api.look.com/looks/123/reports/public.html"
        }
    },
    "actors":{  
        "looker":{  
            "firstName":"Rosa",
            "lastName":"Mart"
        },
        "client":{  
            "email":"XXX.XXX@XXXXXX.com",
            "firstName":"XXX",
            "lastName":"XXX"
        }
    },
    "_id":"123",
    "fields":[  
                {  
        "id":"fence_condition_missing_sections",
        "context":[  
            "Fence Condition"
        ],
        "label":"Missing Sections",
        "type":"choice",
        "value":"None"
    },
        {  
            "id":"photos_landscaped_area",
            "context":[  
                "Landscaping Photos"
            ],
            "label":"Landscaped Area",
            "type":"photo-with-description",
            "value":[  
                {  
                    "description":"Front",
                    "photo":"https://reports-wegolook-com.s3-accelerate.amazonaws.com/looks/123/looker/1.jpg"
                },
                {  
                    "description":"Front entrance ",
                    "photo":"https://reports-wegolook-com.s3-accelerate.amazonaws.com/looks/123/looker/2.jpg"
                }
            ]
        }
    ],
    "jobNumber":"xxx",
    "createdAt":"2018-10-11T22:39:37.223Z",
    "completedAt":"2018-01-27T20:13:49.937Z",
    "inspectedAt":"2018-01-21T23:33:48.718Z",
    "type":"ZZZ-commercial",
    "name":"Commercial"
}'

预期输出：

--------------------------------------------------------------------------------
| ID     | LABEL |  VALUE                                                  | 
--------------------------------------------------------------------------------
|   photos_landscaped_area     |  Landscaped Area |  [{"description":"Front",...}]    |
----------------------------------------------------------------------------
| fence_condition_missing_sections | Missing Sections | None|
----------------------------------------------------------------------------

Answer 1

我将假设您的数据采用每行一个文档的格式，并且出于可读性考虑，您提供了格式化示例。如果不正确，请参阅问题Multi-line JSON file querying in hive 。

当JSON文档的架构不完全规则时，您可以将该列创建为string列，并使用JSON_*函数从中提取值。

首先，您需要为原始数据创建一个表：

CREATE TABLE data (
  fields array<struct<id:string,label:string,value:string>>
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://…'

（如果您对JSON文档中的其他字段不感兴趣，则可以在创建表时忽略它们）

然后创建一个将数据展平的视图：

CREATE VIEW flat_data AS
SELECT
  field.id,
  field.label,
  field.value
FROM data
CROSS JOIN UNNEST(fields) AS f(field)

从该视图中选择应该会为您提供所需的结果。

我怀疑您也在寻找如何从values结构中提取属性的方法，这就是我上面提到的内容：

SELECT
  label,
  JSON_EXTRACT(value, '$.photo') AS photo_urls
FROM flat_data
WHERE id = 'photos_landscaped_area'

在Presto文档中查找所有可用的JSON functions。

如何使用AWS Athena-Presto从NESTED JSON中的特定字段提取数据？

1 个答案: