Question

我之前发过了类似的question。我试图使用以下方法刮取web page

import requests

url = 'https://www.zameen.com/'
res = requests.get(url)
data = res.text
print(data)

它的回复说我要么是BOT，要么是没有启用Javascript。所以，我已经检查但Javascript已启用。所以我尝试了另一种使用假用户代理的方法，其中包含以下代码

from fake_useragent import UserAgent
headers = {}
headers['User-Agent'] = str(ua.chrome)
web_page = requests.get(url,headers=headers)
print(web_page.content)

响应：

b'<!DOCTYPE html>\n\n\t\n\n\t\n\t\n\t\n\n\t\n\t\n\n\t\n\t\n\t\n\n<head>\n<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">\n<meta http-equiv="cache-control" content="max-age=0" />\n<meta http-equiv="cache-control" content="no-cache" />\n<meta http-equiv="expires" content="0" />\n<meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT" />\n<meta http-equiv="pragma" content="no-cache" />\n<meta http-equiv="refresh" content="10; url=/distil_r_captcha.html?Ref=/&amp;distil_RID=053235A2-0030-11E7-8429-B03805AB611E&amp;distil_TID=20170303163950" />\n<script type="text/javascript">\n\t(function(window){\n\t\ttry {\n\t\t\tif (typeof sessionStorage !== \'undefined\'){\n\t\t\t\tsessionStorage.setItem(\'distil_referrer\', document.referrer);\n\t\t\t}\n\t\t} catch (e){}\n\t})(window);\n</script>\n<script type="text/javascript" src="/ga368490.js" defer></script><style type="text/css">#d__fFH{position:absolute;top:-5000px;left:-5000px}#d__fF{font-family:serif;font-size:200px;visibility:hidden}#caexxxzxycbzutyvy{display:none!important}</style></head>\n<body>\n<div id="distil_ident_block">&nbsp;</div>\n</body>\n</html>\n'

它又一次发现我是机器人。所以我检查了我是否可以从网站上获取数据。然后我使用urllib中的robotparser

from urllib import robotparser

req = robotparser.RobotFileParser()
req.set_url(url)
req.read()
print(req.can_fetch('*','https://www.zameen.com/'))

返回：

TRUE # Means I can fetch the data from the website.

有没有办法从这个网页获取数据？感谢

Answer 1

您可以使用BeautifulSoup和Selenium驱动程序。我从您提供的URL获取页面源成功了：

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Firefox() # Could be any other browser you have the drivers for
driver.get('https://zameen.com')
html = driver.page_source
code = BeautifulSoup(html, 'html5lib')
print code

请不要忘记安装bs4和Selenium：

pip install bs4

pip install selenium

无法使用Python抓取网页

1 个答案: