数据框大JSON（嵌套嵌套）

Question

我需要在以下数据框（df3）中对json对象进行数据框处理：

我的代码是：

import pandas as pd 
df =pd.read_json('fetched_tweets.json', orient='records', lines=True)
df2 = pd.DataFrame(df)
df3 = pd.DataFrame.from_dict(df2['entities'])

我也尝试过：

y = df3['entities'].str.lower()

它给了我：

0    NaN
1    NaN
2    NaN
3    NaN
4    NaN
etc .. 

我还通过实现以下代码尝试了this answer：

import json
import pandas as pd
df =pd.read_json('test.json', orient='records', lines=True)
df6=df[['created_at']].join(pd.DataFrame(df['entities'].
apply(json.loads).values.tolist()))
print (df6)

它没有用... 我在这个任务上花了很多时间，而我并不是真正的python专家。请帮忙！

JSON数据示例：

[{"created_at": "Wed Dec 19 22:22:50 +0000 2018", "id": 1075516836656136194, "id_str": "1075516836656136194", "text": "@Yahoo Well your about 16 months late on this bombshell: \nhttps://t.co", "display_text_range": [7, 81], "source": "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>", "truncated": false, "in_reply_to_status_id": 1075512357621850112, "in_reply_to_status_id_str": "1075512357621850112", "in_reply_to_user_id": 19380829, "in_reply_to_user_id_str": "19380829", "in_reply_to_screen_name": "Yahoo", "user": {"id": 1075515640302186497, "id_str": "1075515640302186497", "name": "Dave Wasilla", "screen_name": "DaveWasilla", "location": null, "url": null, "description": null, "translator_type": "none", "protected": false, "verified": false, "followers_count": 0, "friends_count": 0, "listed_count": 0, "favourites_count": 0, "statuses_count": 1, "created_at": "Wed Dec 19 22:18:05 +0000 2018", "utc_offset": null, "time_zone": null, "geo_enabled": false, "lang": "en", "contributors_enabled": false, "is_translator": false, "profile_background_color": "F5F8FA", "profile_background_image_url": "", "profile_background_image_url_https": "", "profile_background_tile": false, "profile_link_color": "1DA1F2", "profile_sidebar_border_color": "C0DEED", "profile_sidebar_fill_color": "DDEEF6", "profile_text_color": "333333", "profile_use_background_image": true, "profile_image_url": "http://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png", "profile_image_url_https": "https://abs.twimg.com/sticky/default_profile_images/default_profile_normal.png", "default_profile": true, "default_profile_image": false, "following": null, "follow_request_sent": null, "notifications": null}, "geo": null, "coordinates": null, "place": null, "contributors": null, "quoted_status_id": 1075238373927895040, "quoted_status_id_str": "1075238373927895040", "quoted_status": {"created_at": "Wed Dec 19 03:56:19 +0000 2018", "id": 1075238373927895040, "id_str": "1075238373927895040", "text": "@mkraju \"Turns out?\" Donald Trump Junior testified his father signed this letter of intent back in September of 201\u2026 https://t.co", "display_text_range": [8, 140], "source": "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>", "truncated": true, "in_reply_to_status_id": 1075236204654850048, "in_reply_to_status_id_str": "1075236204654850048", "in_reply_to_user_id": 39155029, "in_reply_to_user_id_str": "39155029", "in_reply_to_screen_name": "mkraju", "user": {"id": 18005085, "id_str": "18005085", "name": "Mollie", "screen_name": "MZHemingway", "location": "Senior Editor, @FDRLST", "url": "http://thefederalist.com", "description": "I don't share your groupthink and it's unlikely I will be bullied into sharing it. Have a great day.", "translator_type": "none", "protected": false, "verified": true, "followers_count": 215017, "friends_count": 1512, "listed_count": 2724, "favourites_count": 11096, "statuses_count": 88844, "created_at": "Tue Dec 09 22:44:12 +0000 2008", "utc_offset": null, "time_zone": null, "geo_enabled": false, "lang": "en", "contributors_enabled": false, "is_translator": false, "profile_background_color": "000000", "profile_background_image_url": "http://abs.twimg.com/images/themes/theme17/bg.gif", "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme17/bg.gif", "profile_background_tile": false, "profile_link_color": "19CF86", "profile_sidebar_border_color": "000000", "profile_sidebar_fill_color": "000000", "profile_text_color": "000000", "profile_use_background_image": false, "profile_image_url": "http://pbs.twimg.com/profile_images/648645685244510208/HD4ui7lh_normal.jpg", "profile_image_url_https": "https://pbs.twimg.com/profile_images/648645685244510208/HD4ui7lh_normal.jpg", "profile_banner_url": "https://pbs.twimg.com/profile_banners/18005085/1541047835", "default_profile": false, "default_profile_image": false, "following": null, "follow_request_sent": null, "notifications": null}, "geo": null, "coordinates": null, "place": null, "contributors": null, "is_quote_status": false, "extended_tweet": {"full_text": "@mkraju \"Turns out?\" Donald Trump Junior testified his father signed this letter of intent back in September of 2017. Link here (https://t.co) Relevant portion of testimony shown. https://t.co", "display_text_range": [8, 190], "entities": {"hashtags": [], "urls": [{"url": "https://t.co", "expanded_url": "https://www.judiciary.senate.gov/imo/media/doc/Trump%20Jr%20Transcript_redacted.pdf", "display_url": "judiciary.senate.gov/imo/media/doc/\u2026", "indices": [129, 152]}], "user_mentions": [{"screen_name": "mkraju", "name": "Manu Raju", "id": 39155029, "id_str": "39155029", "indices": [0, 7]}], "symbols": [], "media": [{"id": 1075238230591791105, "id_str": "1075238230591791105", "indices": [191, 214], "media_url": "http://pbs.twimg.com/media/DuwDeU9X4AEFRbt.jpg", "media_url_https": "https://pbs.twimg.com/media/DuwDeU9X4AEFRbt.jpg", "url": "https://t.co", "display_url": "pic.twitter.com/VLBpb7YkEr", "expanded_url": "https://twitter.com/MZHemingway/status/1075238373927895040/photo/1", "type": "photo", "sizes": {"small": {"w": 371, "h": 680, "resize": "fit"}, "thumb": {"w": 150, "h": 150, "resize": "crop"}, "large": {"w": 532, "h": 975, "resize": "fit"}, "medium": {"w": 532, "h": 975, "resize": "fit"}}}]}, "extended_entities": {"media": [{"id": 1075238230591791105, "id_str": "1075238230591791105", "indices": [191, 214], "media_url": "http://pbs.twimg.com/media/DuwDeU9X4AEFRbt.jpg", "media_url_https": "https://pbs.twimg.com/media/DuwDeU9X4AEFRbt.jpg", "url": "https://t.co", "display_url": "pic.twitter.com/VLBpb7YkEr", "expanded_url": "https://twitter.com/MZHemingway/status/1075238373927895040/photo/1", "type": "photo", "sizes": {"small": {"w": 371, "h": 680, "resize": "fit"}, "thumb": {"w": 150, "h": 150, "resize": "crop"}, "large": {"w": 532, "h": 975, "resize": "fit"}, "medium": {"w": 532, "h": 975, "resize": "fit"}}}]}}, "quote_count": 62, "reply_count": 73, "retweet_count": 203, "favorite_count": 487, "entities": {"hashtags": [], "urls": [{"url": "https://t.co", "expanded_url": "https://twitter.com/i/web/status/1075238373927895040", "display_url": "twitter.com/i/web/status/1\u2026", "indices": [117, 140]}], "user_mentions": [{"screen_name": "mkraju", "name": "Manu Raju", "id": 39155029, "id_str": "39155029", "indices": [0, 7]}], "symbols": []}, "favorited": false, "retweeted": false, "possibly_sensitive": false, "filter_level": "low", "lang": "en"}, "quoted_status_permalink": {"url": "https://t.co", "expanded": "https://twitter.com/MZHemingway/status/1075238373927895040", "display": "twitter.com/MZHemingway/st\u2026"}, "is_quote_status": true, "quote_count": 0, "reply_count": 0, "retweet_count": 0, "favorite_count": 0, "entities": {"hashtags": [], "urls": [{"url": "https://t.co", "expanded_url": "https://twitter.com/MZHemingway/status/1075238373927895040", "display_url": "twitter.com/MZHemingway/st\u2026", "indices": [58, 81]}], "user_mentions": [{"screen_name": "Yahoo", "name": "Yahoo", "id": 19380829, "id_str": "19380829", "indices": [0, 6]}], "symbols": []}, "favorited": false, "retweeted": false, "possibly_sensitive": false, "filter_level": "low", "lang": "en", "timestamp_ms": "1545258170652"}, {another JSON object},{another object}]

**EDIT**

我可以使用以下代码对实体字符串进行分割：

import json
import pandas as pd
p = 'test.json'
with open(p, 'r') as f:
doc = json.loads(f.read())
df=pd.DataFrame.from_dict(doc[0]['entities']['urls'])

但是它只给我第一行，我需要获取所有行（尽管我不得不提到索引，否则它会给我一个错误）

Answer 1

您可以尝试这个吗？

import json
from pandas.io.json import json_normalize

d = []

with open('json/b.json') as json_data:
    d = json.load(json_data)

df3 = json_normalize(data=d)

Answer 2

因为我嵌套了嵌套值，所以我可以这样解决它：

import pandas as pd
import json
from pandas.io.json import json_normalize
with open('test.json') as data_file:    
data = json.loads(data_file.read())
cols = ['entities', 'urls', 'url', 'expanded_url', 'display_url']
rows = []
for data in data:
    data_id = data['entities']
    criteria = data['entities']['urls']
    for d in criteria:
       rows.append([data_id, criteria.index(d)+1, *list(d.values())[:-1]])
df = pd.DataFrame(rows, columns=cols)

这就是它的样子：

此answer是我的参考

Answer 3

     import pandas as pd
     import json
     from pandas.io.json import json_normalize

     with open('test.json') as data_file: 

     data = json.loads(data_file.read())
     cols = ['entities', 'urls', 'url', 'expanded_url', 'display_url']
     rows = []
     for data in data:
         data_id = data['entities']
         # new json object added
         data_idj = json.loads(data_id)
         criteria = data_idj['entities']['urls']
         for d in criteria:
            rows.append([data_id, criteria.index(d)+1, *list(d.values())[:-1]])
     df = pd.DataFrame(rows, columns=cols)