Question

我正在尝试使用Python的urllib.request下载一些内容。以下命令产生异常：

import urllib.request
print(urllib.request.urlopen("https://fpgroup.foreignpolicy.com/foreign-policy-releases-mayjune-spy-issue/").code)

结果：

...
HTTPError: HTTP Error 403: Forbidden

如果我使用firefox或链接（命令行浏览器）我得到的内容和状态代码为200.如果我使用lynx，很奇怪，我也得到403。

我希望所有方法都能正常工作

同样的方式
成功

为什么不是这样？

Answer 1

该网站最有可能阻止人们抓取他们的网站。您可以通过包含标题信息以及其他内容来在基本级别欺骗它们。有关详细信息，请参见此处

引自：https://docs.python.org/3/howto/urllib2.html#headers

import urllib.parse
import urllib.request

url = 'http://www.someserver.com/cgi-bin/register.cgi'
user_agent = 'Mozilla/5.0 (Windows NT 6.1; Win64; x64)'
values = {'name' : 'Michael Foord',
          'location' : 'Northampton',
          'language' : 'Python' }
headers = { 'User-Agent' : user_agent }

data = urllib.parse.urlencode(values)
data = data.encode('ascii')
req = urllib.request.Request(url, data, headers)
with urllib.request.urlopen(req) as response:
   the_page = response.read()

人们不希望脚本刮掉他们的网站有很多原因。它需要一个带宽。他们不希望人们通过制作一个刮板机器人来获益（金钱方面）。也许他们不希望您复制他们的网站信息。你也可以把它想象成一本书。作者希望人们阅读他们的书籍，但也许他们中的一些人不希望机器人扫描他们的书籍，创建副本，或者机器人可能会总结它。

评论中你问题的第二部分是模糊和广泛的回答，因为有太多自以为是的答案。

Answer 2

我尝试使用此代码，一切正常。

我刚刚在请求中添加了headers。请参阅以下示例：

from urllib.request import Request, urlopen, HTTPError
from time import sleep

def get_url_data(url = ""):
    try:
        request = Request(url, headers = {'User-Agent' :\
            "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2227.0 Safari/537.36"})

        response = urlopen(request)
        data = response.read().decode("utf8")
        return data
    except HTTPError:
        return None

url = "https://fpgroup.foreignpolicy.com/foreign-policy-releases-mayjune-spy-issue/"

for i in range(50):
    d = get_url_data(url)
    if d != None:
        print("Attempt %d was a Success" % i)
    else:
        print("Attempt %d was a Failure" % i)
    sleep(1)

输出：

Attempt 0 was a Success
Attempt 1 was a Success
Attempt 2 was a Success
Attempt 3 was a Success
Attempt 4 was a Success
Attempt 5 was a Success
Attempt 6 was a Success
Attempt 7 was a Success
Attempt 8 was a Success
Attempt 9 was a Success
...
Attempt 42 was a Success
Attempt 43 was a Success
Attempt 44 was a Success
Attempt 45 was a Success
Attempt 46 was a Success
Attempt 47 was a Success
Attempt 48 was a Success
Attempt 49 was a Success

为什么urllib.request.urlopen有时不起作用，但浏览器有效？

2 个答案: