我知道这个问题已被多次询问过。我尝试了几种解决方案，但我无法解决问题。
我有一个大型嵌套JSON文件（1.4GB），我想让它变平，然后将其转换为CSV文件。
JSON结构是这样的：
{
"company_number": "12345678",
"data": {
"address": {
"address_line_1": "Address 1",
"locality": "Henley-On-Thames",
"postal_code": "RG9 1DP",
"premises": "161",
"region": "Oxfordshire"
},
"country_of_residence": "England",
"date_of_birth": {
"month": 2,
"year": 1977
},
"etag": "26281dhge33b22df2359sd6afsff2cb8cf62bb4a7f00",
"kind": "individual-person-with-significant-control",
"links": {
"self": "/company/12345678/persons-with-significant-control/individual/bIhuKnFctSnjrDjUG8n3NgOrl"
},
"name": "John M Smith",
"name_elements": {
"forename": "John",
"middle_name": "M",
"surname": "Smith",
"title": "Mrs"
},
"nationality": "Vietnamese",
"natures_of_control": [
"ownership-of-shares-50-to-75-percent"
],
"notified_on": "2016-04-06"
}
}
我知道使用pandas模块很容易实现，但我不熟悉它。
EDITED
所需的输出应该是这样的：
company_number, address_line_1, locality, country_of_residence, kind,
12345678, Address 1, Henley-On-Thamed, England, individual-person-with-significant-control
请注意，这只是简短版本。输出应该包含所有字段。
答案 0 :(得分：2)
对于您提供的JSON数据，您可以通过解析JSON结构来返回所有叶节点的列表来完成此操作。
这假设您的结构始终一致，如果每个条目可以有不同的字段，请参阅第二种方法。
例如：
import json
import csv
def get_leaves(item, key=None):
if isinstance(item, dict):
leaves = []
for i in item.keys():
leaves.extend(get_leaves(item[i], i))
return leaves
elif isinstance(item, list):
leaves = []
for i in item:
leaves.extend(get_leaves(i, key))
return leaves
else:
return [(key, item)]
with open('json.txt') as f_input, open('output.csv', 'w', newline='') as f_output:
csv_output = csv.writer(f_output)
write_header = True
for entry in json.load(f_input):
leaf_entries = sorted(get_leaves(entry))
if write_header:
csv_output.writerow([k for k, v in leaf_entries])
write_header = False
csv_output.writerow([v for k, v in leaf_entries])
如果您的JSON数据是您提供的格式的条目列表，那么您应该获得如下输出：
address_line_1,company_number,country_of_residence,etag,forename,kind,locality,middle_name,month,name,nationality,natures_of_control,notified_on,postal_code,premises,region,self,surname,title,year
Address 1,12345678,England,26281dhge33b22df2359sd6afsff2cb8cf62bb4a7f00,John,individual-person-with-significant-control,Henley-On-Thames,M,2,John M Smith,Vietnamese,ownership-of-shares-50-to-75-percent,2016-04-06,RG9 1DP,161,Oxfordshire,/company/12345678/persons-with-significant-control/individual/bIhuKnFctSnjrDjUG8n3NgOrl,Smith,Mrs,1977
Address 1,12345679,England,26281dhge33b22df2359sd6afsff2cb8cf62bb4a7f00,John,individual-person-with-significant-control,Henley-On-Thames,M,2,John M Smith,Vietnamese,ownership-of-shares-50-to-75-percent,2016-04-06,RG9 1DP,161,Oxfordshire,/company/12345678/persons-with-significant-control/individual/bIhuKnFctSnjrDjUG8n3NgOrl,Smith,Mrs,1977
如果每个条目可以包含不同（或可能缺少）的字段，那么更好的方法是使用DictWriter。在这种情况下，需要处理所有条目以确定可能fieldnames的完整列表，以便可以写入正确的标头。
import json
import csv
def get_leaves(item, key=None):
if isinstance(item, dict):
leaves = {}
for i in item.keys():
leaves.update(get_leaves(item[i], i))
return leaves
elif isinstance(item, list):
leaves = {}
for i in item:
leaves.update(get_leaves(i, key))
return leaves
else:
return {key : item}
with open('json.txt') as f_input:
json_data = json.load(f_input)
# First parse all entries to get the complete fieldname list
fieldnames = set()
for entry in json_data:
fieldnames.update(get_leaves(entry).keys())
with open('output.csv', 'w', newline='') as f_output:
csv_output = csv.DictWriter(f_output, fieldnames=sorted(fieldnames))
csv_output.writeheader()
csv_output.writerows(get_leaves(entry) for entry in json_data)
答案 1 :(得分：2)
您可以使用pandas库json_normalize函数来展平该结构，然后根据需要对其进行处理。例如：
import pandas as pd
import json
raw = """[{
"company_number": "12345678",
"data": {
"address": {
"address_line_1": "Address 1",
"locality": "Henley-On-Thames",
"postal_code": "RG9 1DP",
"premises": "161",
"region": "Oxfordshire"
},
"country_of_residence": "England",
"date_of_birth": {
"month": 2,
"year": 1977
},
"etag": "26281dhge33b22df2359sd6afsff2cb8cf62bb4a7f00",
"kind": "individual-person-with-significant-control",
"links": {
"self": "/company/12345678/persons-with-significant-control/individual/bIhuKnFctSnjrDjUG8n3NgOrl"
},
"name": "John M Smith",
"name_elements": {
"forename": "John",
"middle_name": "M",
"surname": "Smith",
"title": "Mrs"
},
"nationality": "Vietnamese",
"natures_of_control": [
"ownership-of-shares-50-to-75-percent"
],
"notified_on": "2016-04-06"
}
}]"""
data = json.loads(raw)
data = pd.json_normalize(data)
print(data.to_csv())
哪个给你：
,company_number,data.address.address_line_1,data.address.locality,data.address.postal_code,data.address.premises,data.address.region,data.country_of_residence,data.date_of_birth.month,data.date_of_birth.year,data.etag,data.kind,data.links.self,data.name,data.name_elements.forename,data.name_elements.middle_name,data.name_elements.surname,data.name_elements.title,data.nationality,data.natures_of_control,data.notified_on
0,12345678,Address 1,Henley-On-Thames,RG9 1DP,161,Oxfordshire,England,2,1977,26281dhge33b22df2359sd6afsff2cb8cf62bb4a7f00,individual-person-with-significant-control,/company/12345678/persons-with-significant-control/individual/bIhuKnFctSnjrDjUG8n3NgOrl,John M Smith,John,M,Smith,Mrs,Vietnamese,['ownership-of-shares-50-to-75-percent'],2016-04-06
答案 2 :(得分：2)
这是一个比较老的问题，但是我整夜都在努力争取类似情况的满意结果，于是我想到了：
import json
import pandas
def cross_join(left, right):
return left.assign(key=1).merge(right.assign(key=1), on='key', how='outer').drop('key', 1)
def json_to_dataframe(data_in):
def to_frame(data, prev_key=None):
if isinstance(data, dict):
df = pandas.DataFrame()
for key in data:
df = cross_join(df, to_frame(data[key], prev_key + '.' + key))
elif isinstance(data, list):
df = pandas.DataFrame()
for i in range(len(data)):
df = pandas.concat([df, to_frame(data[i], prev_key)])
else:
df = pandas.DataFrame({prev_key[1:]: [data]})
return df
return to_frame(data_in)
if __name__ == '__main__':
with open('somefile') as json_file:
json_data = json.load(json_file)
df = json_to_dataframe(json_data)
df.to_csv('data.csv', mode='w')
说明：
cross_join 函数是我发现做笛卡尔乘积的一种巧妙方法。 （信用：here）
json_to_dataframe 函数使用熊猫数据框执行逻辑。就我而言，json是深层嵌套的，我想将字典 key：value对拆分成列，但是我想将 lists转换成列的行- -因此，concat –然后与上层交叉连接，从而乘以记录数，以便列表中的每个值都有自己的行，而前面的列是相同的。
递归会创建与下面的堆栈交叉连接的堆栈，直到返回最后一个堆栈。
然后使用表格式的数据框，使用“ df.to_csv（）” 数据框对象方法很容易将其转换为CSV。
这应该与深度嵌套的JSON一起使用，能够通过上述逻辑将其全部标准化为行。
我希望有一天能对某人有所帮助。只是想回馈这个很棒的社区。
-------------------------------------------- -------------------------------------------------- -------------------------------------
最新编辑：新解决方案
我要回到这个问题，因为dataframe选项还可以工作，但应用程序花了几分钟的时间来解析不太大的JSON数据。因此，我想到了要做数据框的工作，但我自己做
：from copy import deepcopy
import pandas
def cross_join(left, right):
new_rows = []
for left_row in left:
for right_row in right:
temp_row = deepcopy(left_row)
for key, value in right_row.items():
temp_row[key] = value
new_rows.append(deepcopy(temp_row))
return new_rows
def flatten_list(data):
for elem in data:
if isinstance(elem, list):
yield from flatten_list(elem)
else:
yield elem
def json_to_dataframe(data_in):
def flatten_json(data, prev_heading=''):
if isinstance(data, dict):
rows = [{}]
for key, value in data.items():
rows = cross_join(rows, flatten_json(value, prev_heading + '.' + key))
elif isinstance(data, list):
rows = []
for i in range(len(data)):
[rows.append(elem) for elem in flatten_list(flatten_json(data[i], prev_heading))]
else:
rows = [{prev_heading[1:]: data}]
return rows
return pandas.DataFrame(flatten_json(data_in))
if __name__ == '__main__':
json_data = {
"id": "0001",
"type": "donut",
"name": "Cake",
"ppu": 0.55,
"batters":
{
"batter":
[
{"id": "1001", "type": "Regular"},
{"id": "1002", "type": "Chocolate"},
{"id": "1003", "type": "Blueberry"},
{"id": "1004", "type": "Devil's Food"}
]
},
"topping":
[
{"id": "5001", "type": "None"},
{"id": "5002", "type": "Glazed"},
{"id": "5005", "type": "Sugar"},
{"id": "5007", "type": "Powdered Sugar"},
{"id": "5006", "type": "Chocolate with Sprinkles"},
{"id": "5003", "type": "Chocolate"},
{"id": "5004", "type": "Maple"}
]
}
df = json_to_dataframe(json_data)
print(df)
输出：
id type name ppu batters.batter.id batters.batter.type topping.id topping.type
0 0001 donut Cake 0.55 1001 Regular 5001 None
1 0001 donut Cake 0.55 1001 Regular 5002 Glazed
2 0001 donut Cake 0.55 1001 Regular 5005 Sugar
3 0001 donut Cake 0.55 1001 Regular 5007 Powdered Sugar
4 0001 donut Cake 0.55 1001 Regular 5006 Chocolate with Sprinkles
5 0001 donut Cake 0.55 1001 Regular 5003 Chocolate
6 0001 donut Cake 0.55 1001 Regular 5004 Maple
7 0001 donut Cake 0.55 1002 Chocolate 5001 None
8 0001 donut Cake 0.55 1002 Chocolate 5002 Glazed
9 0001 donut Cake 0.55 1002 Chocolate 5005 Sugar
10 0001 donut Cake 0.55 1002 Chocolate 5007 Powdered Sugar
11 0001 donut Cake 0.55 1002 Chocolate 5006 Chocolate with Sprinkles
12 0001 donut Cake 0.55 1002 Chocolate 5003 Chocolate
13 0001 donut Cake 0.55 1002 Chocolate 5004 Maple
14 0001 donut Cake 0.55 1003 Blueberry 5001 None
15 0001 donut Cake 0.55 1003 Blueberry 5002 Glazed
16 0001 donut Cake 0.55 1003 Blueberry 5005 Sugar
17 0001 donut Cake 0.55 1003 Blueberry 5007 Powdered Sugar
18 0001 donut Cake 0.55 1003 Blueberry 5006 Chocolate with Sprinkles
19 0001 donut Cake 0.55 1003 Blueberry 5003 Chocolate
20 0001 donut Cake 0.55 1003 Blueberry 5004 Maple
21 0001 donut Cake 0.55 1004 Devil's Food 5001 None
22 0001 donut Cake 0.55 1004 Devil's Food 5002 Glazed
23 0001 donut Cake 0.55 1004 Devil's Food 5005 Sugar
24 0001 donut Cake 0.55 1004 Devil's Food 5007 Powdered Sugar
25 0001 donut Cake 0.55 1004 Devil's Food 5006 Chocolate with Sprinkles
26 0001 donut Cake 0.55 1004 Devil's Food 5003 Chocolate
27 0001 donut Cake 0.55 1004 Devil's Food 5004 Maple
按照以上所述， cross_join 函数的功能与数据框解决方案中的功能几乎相同，但是没有数据框，因此速度更快。
我添加了 flatten_list 生成器，因为我想确保JSON数组都很好并且被扁平化，然后提供为单个字典列表，其中包含一次迭代中的前一个键，然后分配给每个列表的值。在这种情况下，这几乎模仿了 pandas.concat 行为。
主函数 json_to_dataframe 中的逻辑与以前相同。要做的所有更改就是将数据框所执行的操作作为编码功能。
此外，在数据框解决方案中，我没有将前一个标题附加到嵌套对象上，但是除非您100％确定列名没有冲突，否则它是强制性的。
我希望这会有所帮助：）。
答案 3 :(得分：0)
参考 Bogdan Mircea 的回答，
代码几乎达到了我的目的！ 但是每当它遇到嵌套 json 中的空列表时，它就会返回一个空数据框。
您可以通过将其放入代码中来轻松解决此问题
elif isinstance(data, list):
rows = []
if(len(data) != 0):
for i in range(len(data)):
[rows.append(elem) for elem in flatten_list(flatten_json(data[i], prev_heading))]
else:
data.append(None)
[rows.append(elem) for elem in flatten_list(flatten_json(data[0], prev_heading))]