Question

代码如下：

from requests import get
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36'}

url = 'https://business.inquirer.net/category/latest-stories/page/10'
response = get(url)
print(response.text[:500])
html_soup = BeautifulSoup(response.text, 'html.parser')
type(html_soup)

这就是我得到的结果：

<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

我已经读到放置标头可以解决该错误，但是我尝试放置在检查站点时从devtool复制的标头，但不能解决我的问题请帮助我

Answer 1

您不会在任何地方使用标头变量，因此，您不会在请求中传递它。您可以使用以下代码来做到这一点：

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen

siteurl = "https://business.inquirer.net/category/latest-stories/page/10"
hdr = {'User-Agent': 'Mozilla/5.0'}
req = Request(siteurl,headers=hdr)
page = urlopen(req)
soup = BeautifulSoup(page)
print(soup)

Answer 2

当尝试使用BeautifulSoap网站从该网站中抓取数据时，不会显示其数据。

尝试时：

from bs4 import BeautifulSoup
from urllib import urlopen

url = "https://business.inquirer.net/category/latest-stories/page/10"

open_page = urlopen(url)
source = BeautifulSoup(open_page,"html.parser")

print source

您将看到类似于：

的行

<p>The owner of this website (business.inquirer.net) has banned your access based on your browser's signature (4af0dedd3eebcb40-ua48).</p>

因此不要尝试使用BeautifulSoap来做到这一点。使用Selenium更容易。

from selenium import webdriver


options = webdriver.ChromeOptions()
driver=webdriver.Chrome(chrome_options=options, executable_path=r'your driver path')
driver.get('https://business.inquirer.net/category/latest-stories/page/10')

x = driver.find_elements_by_css_selector("div[id='ch-ls-head']")


for a in x:
  print a.text
driver.close()

输出：

TAXATION
DOF clarifies: Rice tariffication law takes effect on March 5
FEBRUARY 19, 2019 BY:  BEN O. DE VERA
BANKS
HSBC reports net profit at $12.6B in 2018
FEBRUARY 19, 2019
CURRENCIES
Asian shares gain on hopes for progress on China-US trade
FEBRUARY 19, 2019
ECONOMY
Amro sees higher PH growth in 2019 on easing inflation, infra boost
FEBRUARY 19, 2019 BY:  BEN O. DE VERA
TELECOMMUNICATIONS
Poe to DICT: Stop ‘dilly-dallying’ over 3rd telco project
FEBRUARY 19, 2019 BY:  CHRISTIA MARIE RAMOS
SOCIAL SECURITY
SSS contribution collections grow by P22.19B in 2018
FEBRUARY 18, 2019 BY:  CHRISTIA MARIE RAMOS
STOCKS
World stocks mixed ahead of further China-US trade talks
FEBRUARY 18, 2019
TRADE
Rice tariffication starts on March 3
FEBRUARY 18, 2019 BY:  BEN O. DE VERA
AGRICULTURE/AGRIBUSINESS
NFA-Bohol workers wear black to mourn ‘death of the rice industry’
FEBRUARY 18, 2019 BY:  LEO UDTOHAN
BONDS
Treasury: RTBs to be sold to individual investors online in Q1
FEBRUARY 18, 2019 BY:  BEN O. DE VERA

Answer 3

为我工作

from bs4 import BeautifulSoup
import urllib.request 
response = urllib.request.urlopen('https://business.inquirer.net/category/latest-stories/page/10') 
html = response.read()
soup = BeautifulSoup(html,"html5lib")
text = soup.get_text(strip=True)
print (text)

Answer 4

尝试包含标头，许多网站会阻止没有标头的请求：

r = requests.get(url, headers=...)

查看请求文档以获取更多信息：http://docs.python-requests.org/en/master/user/quickstart/

我需要获取新闻文章数据。我正在使用来自python的请求/获取，但出现此错误：403禁止

4 个答案: