使用Pandas通过特定字段将JSON转换为CSV

Question

我目前正在尝试使用Pandas将JSON文件转换为CSV文件。

我现在使用的代码能够将JSON转换为CSV文件。

import pandas as pd
json_data = pd.read_json("out1.json")
from pandas.io.json import json_normalize
df = json_normalize(json_data["events"])
df.to_csv("out.csv)

这是我的JSON文件：

{
  "events": [
    {
      "raw": "{\"level\": \"INFO\", \"message\": \"Disabled camera with QR scan on  by 80801234 at Area A\n\"}",
      "logtypes": [
        "json"
      ],
      "timestamp": 1537190572023,
      "unparsed": null,
      "logmsg": "{\"level\": \"INFO\", \"message\": \"Disabled camera with QR scan on  by 80801234 at Area A\n\"}",
      "id": "c77afb4c-ba7c-11e8-8000-12b233ae723a",
      "tags": [
        "INFO"
      ],
      "event": {
        "json": {
          "message": "Disabled camera with QR scan on  by 80801234 at Area A\n",
          "level": "INFO"
        },
        "http": {
          "clientHost": "116.197.237.29",
          "contentType": "text/plain; charset=UTF-8"
        }
      }
    },
    {
      "raw": "{\"level\": \"INFO\", \"message\": \"Employee number saved successfully.\"}",
      "logtypes": [
        "json"
      ],
      "timestamp": 1537190528619,
      "unparsed": null,
      "logmsg": "{\"level\": \"INFO\", \"message\": \"Employee number saved successfully.\"}",
      "id": "ad9c0175-ba7c-11e8-803d-12b233ae723a",
      "tags": [
        "INFO"
      ],
      "event": {
        "json": {
          "message": "Employee number saved successfully.",
          "level": "INFO"
        },
        "http": {
          "clientHost": "116.197.237.29",
          "contentType": "text/plain; charset=UTF-8"
        }
      }
    }
  ]
}

但是我想要的只是JSON文件中的一些字段（时间戳，级别，消息），而不是全部。

我尝试了多种方法：

df = json_normalize(json_data["timestamp"]) // gives a KeyError on 'timestamp'

df = json_normalize(json_data, 'timestamp', ['event', 'json', ['level', 'message']]) // TypeError: string indices must be integers

我在哪里弄错了？

Answer 1

我不认为json_normalize打算用于此特定方向。我可能是错的，但是从文档中看，规范化的意思是“处理每个词典中的列表”。

假设data是

data = json.load(open('out1.json'))['events']

查看第一个条目

data[0]['timestamp']

1537190572023

json_normalize希望这是一个列表

[{'timestamp': 1537190572023}]

创建增强型data2

我实际上不推荐这种方法。
如果我们相应地创建data2：

data2 = [{**d, **{'timestamp': [{'timestamp': d['timestamp']}]}} for d in data]

我们可以使用json_normalize

json_normalize(
    data2, 'timestamp',
    [['event', 'json', 'level'], ['event', 'json', 'message']]
)

       timestamp event.json.level                                 event.json.message
0  1537190572023             INFO  Disabled camera with QR scan on  by 80801234 a...
1  1537190528619             INFO                Employee number saved successfully.

---

理解力

我认为这样做更简单

pd.DataFrame([
    (d['timestamp'],
     d['event']['json']['level'],
     d['event']['json']['message'])
    for d in data
], columns=['timestamp', 'level', 'message'])

       timestamp level                                            message
0  1537190572023  INFO  Disabled camera with QR scan on  by 80801234 a...
1  1537190528619  INFO                Employee number saved successfully.

---

json_normalize

但没有花哨的论点

json_normalize(data).pipe(
    lambda d: d[['timestamp']].join(
        d.filter(like='event.json')
    )
)

       timestamp event.json.level                                 event.json.message
0  1537190572023             INFO  Disabled camera with QR scan on  by 80801234 a...
1  1537190528619             INFO                Employee number saved successfully.