我试图从纽约时报(纽约时报)的文章中获取一些数据,当我执行以下代码时,它给了我一个我不熟悉的错误,我在google上搜索并从stackoverflow中查看了之前的答案,但不明白我的问题。 谁能告诉我如何解决我的错误。 提前谢谢!
代码:
from nytimesarticle import articleAPI
api = articleAPI('a0de895aa110431eb2344303c7105a9f')
articles = api.search( q = 'Obama',
fq = {'headline':'Obama', 'source':['Reuters','AP', 'The New York Times']},
begin_date = 20111231 )
def parse_articles(articles):
'''
This function takes in a response to the NYT api and parses
the articles into a list of dictionaries
'''
news = []
for i in articles['response']['docs']:
dic = {}
dic['id'] = i['_id']
if i['abstract'] is not None:
dic['abstract'] = i['abstract'].encode("utf8")
dic['headline'] = i['headline']['main'].encode("utf8")
dic['desk'] = i['news_desk']
dic['date'] = i['pub_date'][0:10] # cutting time of day.
dic['section'] = i['section_name']
if i['snippet'] is not None:
dic['snippet'] = i['snippet'].encode("utf8")
dic['source'] = i['source']
dic['type'] = i['type_of_material']
dic['url'] = i['web_url']
dic['word_count'] = i['word_count']
# locations
locations = []
for x in range(0,len(i['keywords'])):
if 'glocations' in i['keywords'][x]['name']:
locations.append(i['keywords'][x]['value'])
dic['locations'] = locations
# subject
subjects = []
for x in range(0,len(i['keywords'])):
if 'subject' in i['keywords'][x]['name']:
subjects.append(i['keywords'][x]['value'])
dic['subjects'] = subjects
news.append(dic)
return(news)
def get_articles(date,query):
'''
This function accepts a year in string format (e.g.'1980')
and a query (e.g.'Amnesty International') and it will
return a list of parsed articles (in dictionaries)
for that year.
'''
all_articles = []
for i in range(0,100): #NYT limits pager to first 100 pages. But rarely will you find over 100 pages of results anyway.
articles = api.search(q = query,
fq = {'source':['Reuters','AP', 'The New York Times']},
begin_date = date + '0101',
end_date = date + '1231',
sort='oldest',
page = str(i))
articles = parse_articles(articles)
all_articles = all_articles + articles
return(all_articles)
Amnesty_all = []
for i in range(1980,2014):
print 'Processing' + str(i) + '...'
Amnesty_year = get_articles(str(i),'Amnesty International')
Amnesty_all = Amnesty_all + Amnesty_year
import csv
keys = Amnesty_all[0].keys()
with open('amnesty-mentions.csv', 'wb') as output_file:
dict_writer = csv.DictWriter(output_file, keys)
dict_writer.writeheader()
dict_writer.writerows(Amnesty_all)
在终端中运行时产生的错误:
rakesh-chinta@rakeshchinta-VirtualBox:~$ cd Desktop
rakesh-chinta@rakeshchinta-VirtualBox:~/Desktop$ python nyt.py
Processing1980...
Traceback (most recent call last):
File "nyt.py", line 66, in <module>
Amnesty_year = get_articles(str(i),'Amnesty International')
File "nyt.py", line 59, in get_articles
articles = parse_articles(articles)
File "nyt.py", line 14, in parse_articles
for i in articles['response']['docs']:
KeyError: 'response'
rakesh-chinta@rakeshchinta-VirtualBox:~/Desktop$ python nyt.py
Processing1980...
Traceback (most recent call last):
File "nyt.py", line 66, in <module>
Amnesty_year = get_articles(str(i),'Amnesty International')
File "nyt.py", line 59, in get_articles
articles = parse_articles(articles)
File "nyt.py", line 14, in parse_articles
for i in articles['response']['docs']:
KeyError: 'response'
答案 0 :(得分:0)
api.search返回不期望的结果。它的代码:
<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
android:id="@+id/activity_main"
android:layout_width="match_parent"
android:layout_height="match_parent"
android:paddingBottom="@dimen/activity_vertical_margin"
android:paddingLeft="@dimen/activity_horizontal_margin"
android:paddingRight="@dimen/activity_horizontal_margin"
android:paddingTop="@dimen/activity_vertical_margin"
tools:context=".MainActivity">
<TextView
android:id="@+id/txtOut"
android:layout_width="match_parent"
android:layout_height="wrap_content"
android:textSize="17sp"
android:textColor="@android:color/background_dark"
android:scrollbars="vertical"
android:isScrollContainer="true"
android:layout_alignParentBottom="true"
android:layout_alignParentLeft="true"
android:layout_alignParentStart="true"
android:textAlignment="center" />
<ListView
android:layout_width="match_parent"
android:layout_height="match_parent"
android:layout_alignParentLeft="true"
android:layout_alignParentStart="true"
android:id="@+id/songListView"
android:layout_above="@+id/txtIn" />
</RelativeLayout>
因此,只有当api“http://api.nytimes.com/svc/search/v2/articlesearch”返回正确的响应并且响应具有正确的json正文时,您才能正确运行代码。
异常是KeyError,所以返回对象就是dict。你可能想检查一下:
r = requests.get(url)
return r.json()
和
In [8]: print articles.keys()
Out[8]: [u'status', u'response', u'copyright']
如果没有,我猜nyt api可能无法在文章['status']!='OK'时填写响应,您可能需要处理此意外状态并重试。
答案 1 :(得分:0)
在循环中引入time.sleep(1)
import time
像这样:
articles = parse_articles(articles)
all_articles = all_articles + articles
time.sleep(1)
return(all_articles)
此错误是由于程序包内部使用的API每秒最多限制5个请求。