urlopen HTTP错误

Question

我尝试使用BeautifulSoup打开gutenberg项目的页面

import urllib2
from bs4 import BeautifulSoup

url = "http://www.gutenberg.org/files/54801/54801-h/54801-h.htm"
page = urllib2.urlopen(url)
soup_packtpage=BeautifulSoup(page)

print(soup_packtpage)

我在cloud9工作。我有以下错误：

Traceback (most recent call last):
File "soup.py", line 5, in <module>
page = urllib2.urlopen(url)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 410, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 448, in error
return self._call_chain(*args)
File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 531, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

有什么问题？

Answer 1

import cookielib
import urllib2

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12;rv:50.0) Gecko/20100101 Firefox/50.0'}
cookie = cookielib.CookieJar()
handler = urllib2.HTTPCookieProcessor(cookie)
opener = urllib2.build_opener(handler)

request = urllib2.Request(url = "http://www.gutenberg.org/files/54801/54801-h/54801-h.htm", headers=headers)
page = opener.open(request).read()

尝试请求并添加标头。它适用于Python 2.7.13

Answer 2

您应该尝试使用请求包。

这在python 3.6

中适合我

import requests
from bs4 import BeautifulSoup as bs4

url = "http://www.gutenberg.org/files/54801/54801-h/54801-h.htm"
r = requests.get(url)
#page = urllib3.urlopen(url)
soup_packtpage = bs4(r.text, 'html.parser')

print(soup_packtpage)

paragrapghs = soup_packtpage.findAll("p")
print(paragrapghs)

f = open("guttenberg_book.html", 'a', encoding="utf-8")
f.write(str(paragrapghs))
f.close()

我添加了一个使用BS4的打印段落让你开始..这只输出书籍文本.. :)