Question

当我使用python urllib.request解析URL时，我得到了403 Forbidden。这是代码：

import urllib.request
url='https://www.genecards.org/cgi-bin/carddisp.pl?gene=ERBB2'
headers=('User-Agent','Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36')
opener=urllib.request.build_opener()
opener.addheaders=[headers]
urllib.request.install_opener(opener)
data=urllib.request.urlopen(url).read().decode('utf-8')
print(data)

然后我得到一个错误：

Traceback (most recent call last):
  File "/Users/zhangqing/Documents/Yanpu/ERBB2 Gene.py", line 22, in <module>
    data=urllib.request.urlopen(url).read().decode('utf-8')
  File "/Users/zhangqing/anaconda/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/zhangqing/anaconda/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/Users/zhangqing/anaconda/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Users/zhangqing/anaconda/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/Users/zhangqing/anaconda/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Users/zhangqing/anaconda/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

当我尝试来自python的请求时，代码为：

import requests
import re
from requests.exceptions import RequestException

def get_page(url):
headers={'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36'}
try:
    res=requests.get(url,headers=headers)
    if res.status_code==200:
        return res.text
except RequestException:
    return None

html=get_page('https://www.genecards.org/cgi-bin/carddisp.pl?gene=ERBB2')
print(html)

我有一个类似这样的html：

请求失败。封装事件ID：461001240193404751-556133389381208009

这不是网页的真实源代码。那我该怎么做才能改善代码？

Answer 1

该网页正在使用Incapsula，而Incapsula发现您正在使用漫游器。有关一些可能的解决方法，请参见this question，或者尝试找到genecard.com的公共API（如果有的话）。

无法使用python请求获取真正的html

1 个答案: