Question

我正在使用python的Requests库从BestBuy Products API下载一些数据，我想将它们存储到pandas数据帧中。

类似的东西：

results = requests.get(url1, 
                  params={'paramStuff'}, 
                  headers={'User-Agent': ua})
products = json.loads(results.text)

通过服务信息获取了很多不同的字段，因此我只针对JSON中我想要的特定字段：

products['products']

我有：

[{'details':[{'name': 'Name of Feature', 'value':'Value Of Feature'},
             {'name': 'Name of Other Feature', 'value':'Value Of Other 
              Feature'}, ...],
   'ProductId': 'Id Of Product 1',
   'Some Other Field': 'Some Other Field Value'},
 {same structure as above for other product}, {etc}]

因此，您可以看到它类似于字典列表，而这些字典又包含字典列表。要突出显示 - 详细信息dict可以有各种名称组合列表：值（产品名称也不同）。

关于如何使用这种格式进入这种结构的任何想法：

+-----------+-------------------+-------------------+-------------------+------------------+
| ProductID | Name of Feature 1 | Name of Feature 2 | Name Of Feature 3 | Some Other Field |
+-----------+-------------------+-------------------+-------------------+------------------+
| Product 1 | Value             | NULL              | Value             | Value            |
| Product 2 | NULL              | Value             | Value             | Value            |
+-----------+-------------------+-------------------+-------------------+------------------+

到目前为止，我只能做到这样的事情：

+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+
| ProductID |                                                              Details                                                              | Some Other Field |
+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+
| Product 1 | [{'name': 'Name of Feature', 'value':'Value Of Feature'},{'name': 'Name of Other Feature', 'value':'Value Of Other Feature'},...] | Value 1          |
| Product 2 | [{'name': 'Name of Feature', 'value':'Value Of Feature'},{'name': 'Name of Other Feature', 'value':'Value Of Other Feature'},...] | Value 2          |
+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+

Answer 1

好的，我最终开发了一种手动解析嵌套字段的方法。没有弄清楚是否有任何简单的方法。仅供参考，用于解析BestBuy Products API的响应，以防有人发现它有用。

#first build the pandas DF shown in question
df = pd.io.json.json_normalize(products)

#fields which are not nested and not require parsing
fields = ['sku', 'name', 'regularPrice', 'manufacturer']
#nested field is called 'details', as mentioned can have a lot of different subfields
featureFields = []


#first build a list which will have all potential features from the nested field
for i in range(0,len(df)):
    row = df.iloc[i]
    for detail in row['details']:
        featureFields.append(detail['name'].split('>', 1)[-1])

#make a list unique
featureFields = set(featureFields)      
fields = set(fields)

#now we go over each record in dataframe and parse nested field to a dict
records = []

for i in range(0,len(df)):
    row = df.iloc[i]
    record = dict.fromkeys(fields)
    record['name'] = row['name']
    record['regularPrice'] = row['regularPrice']
    record['manufacturer'] = row['manufacturer']
    record['sku'] = row['sku']
    for detail in row['details']:
        record[detail['name'].split('>', 1)[-1]] = detail['value'].split('>', 1)[-1]
    records.append(record)

#finally we have not nested list of dictionaries with records
dfFinal = pd.DataFrame.from_dict(records)

将具有各种字段列表的JSON响应转换为Pandas数据帧

1 个答案: