Question

我正在尝试使用代理爬取http网站，并且当我尝试提取文本时，它显示为“网页已阻止”。我该如何避免这个错误？

我的代码如下

url = "http://campanulaceae.myspecies.info/"

proxy_dict = { 
          'http'  : "174.138.54.49:8080", 
          'https' : "174.138.54.49:8080"
        }

page = requests.get(url, proxies=proxy_dict)
soup = BeautifulSoup(page.text,'html.parser')
print(soup)

当我尝试从网站输出文本时，我的输出低于输出。

<html>
<head>
<title>Web Page Blocked</title>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
<meta content="NO-CACHE" http-equiv="PRAGMA"/>
<meta content="initial-scale=1.0" name="viewport"/>
........
<body bgcolor="#e7e8e9">
<div id="content">
<h1>Web Page Blocked</h1>
<p>Access to the web page you were trying to visit has been blocked in accordance with company policy. Please contact your system administrator if you believe this is in error.</p>

Answer 1

因为您没有为请求标头指定用户代理。网站经常会阻止来自类似机器人的资源的请求。像这样尝试：

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.106 Safari/537.36'}

page = requests.get(url, headers=headers, proxies=proxy_dict)

使用http进行网页抓取显示“网页被阻止”

1 个答案: