Question

我从API中提取了一个JSON对象。我将JSON数据拉入python字典。我现在发现很难从字典中提取数据，因为它是嵌套的并且有子列表和子字典。为了更好地理解所提取数据的性质，我尝试了下面的内容：

url = "https://api.xyz.com/v11/api.json?KEY=abc&LOOKUP=bbb"
response = requests.get(url)
data = response.json()
print (type(data))
print(data.keys())
print (type(data['Results']))
print (len(data['Results']))
print (type(data['Results'][0]))
print (data['Results'][0].keys())
print (type(data['Results'][0]['Result']))
print ((data['Results'][0]['Result'].keys()))
print (type((data['Results'][0]['Result']['Paths'])))
print (len((data['Results'][0]['Result']['Paths'])))
print (type((data['Results'][0]['Result']['Paths'][0])))
print ((data['Results'][0]['Result']['Paths'][0].keys()))
print (type(data['Results'][0]['Result']['Paths'][0]['Technologies']))
print (len(data['Results'][0]['Result']['Paths'][0]['Technologies']))
print ((data['Results'][0]['Result']['Paths'][0]['Technologies'][8].keys()))
print (data['Results'][0]['Result']['Paths'][0]['Technologies'][8]['Tag'])

从上面我得到了以下输出：

<class 'dict'>
dict_keys(['Results', 'Errors'])
<class 'list'>
1
<class 'dict'>
dict_keys(['Lookup', 'LastIndexed', 'FirstIndexed', 'Meta', 'Result'])
<class 'dict'>
dict_keys(['Paths', 'IsDB', 'Spend'])
<class 'list'>
9
<class 'dict'>
dict_keys(['LastIndexed', 'Technologies', 'Domain', 'SubDomain', 'FirstIndexed', 'Url'])
<class 'list'>
77
dict_keys(['FirstDetected', 'Name', 'LastDetected', 'Categories', 'Description', 'IsPremium', 'Tag', 'Link'])
cdn

从其他迭代开始，我知道根据我在“路径”之后选择的列表项，我可以得到“技术”的变化列表长度，范围从5到100。我特别感兴趣的是获取“Tag”== A的所有技术的列表。我希望能够创建一个包含所有条目的所有上层信息的表，其中包含'Tag'== A.理想情况下，我想在CSV文件中获取此信息。我查看了Pandas dataframe from nested dictionary和Create a dictionary with list comprehension in Python以及Construct pandas DataFrame from items in nested dictionary，但在访问列表时感到困惑（特别是在“路径”之后）。

到目前为止，我所拥有的代码是一个简单的数据转储到CSV中，完全没用，因为所有数据都进入一个单元格并且根本不可用。

Answer 1

所以我找到了一种方法来做到这一点。代码如下。

import json
import requests
import pandas

domaindict = {}
tech_info = {}
tag = 'mobile'

'''For every domain, pulls the api response, stores in dictionary, and gets dataframe
with technology info by calling the get_paths and set_df functions'''

def get_info(d):
    url = "https://api.xyz.com/v11/api.json?KEY=abc&LOOKUP={}".format(d)
    response = requests.get(url)
    data = response.json()
    domaindict[d] = data
    paths = get_paths(d)
    final_info = set_df(paths)
    return final_info

'''Gets the list of paths for a domain'''        
def get_paths(d):
    paths = domaindict[d]['Results'][0]['Result']['Paths']
    return paths

'''Sets up dataframe to get info at the technology level. Loops through the technology list and path lists.'''
def set_df(paths):
    df_m = pandas.DataFrame({'Domain':[],'Url':[],'Subdomain':[],'Technology Name':[],'Technology Description':[],'Technology Tag':[]})
    for path in paths:
        domain_name = path['Domain']
        url = path['Url']
        subdomain = path['SubDomain']
        techs = path['Technologies']
        for tech in techs: 
            tech_name = tech['Name']
            tech_desc = tech['Description']
            tech_tag = tech['Tag']
            df =  pandas.DataFrame({'Domain':[domain_name], 'Url':[url], 'Subdomain':[subdomain], 
                                    'Technology Name':[tech_name],
                                     'Technology Description': [tech_desc],
                                     'Technology Tag': [tech_tag]})
            df_m = df_m.append(df)
        return df_m


'''loops through the csv file with list of domain names, calls the get_info function and saves dataframe with technology info for
every domain in one file'''

read_domains = pandas.read_excel('domain.xlsx', header = None)
df_f = pandas.DataFrame()
for d in read_domains.values: 
    print(d[0])
    df_i = get_info(d[0])
    df_f = df_f.append(df_i)


with pandas.ExcelWriter('mobile.xlsx') as w:
    df_f.to_excel(w,'mobile')

创建一种简单的方法来读取保存在复杂的嵌套python字典中的JSON对象

1 个答案: