Question

我试图抓取的文字是来自

的第123次会议的标题

https://www.bcb.gov.br/en/#!/c/copomstatements/1724

为此，我使用此代码

import urllib.request           #get the HTML page from url 
import urllib.error

from bs4 import BeautifulSoup


# set page to read
with urllib.request.urlopen('https://www.bcb.gov.br/en/#!/c/copomstatements/1724') as response:
   page = response.read()

# parse the html using beautiful soup and store in variable `soup`
soup = BeautifulSoup(page, "html.parser")
print(soup)

# Inspect: <h3 class="BCTituloPagina ng-binding">123rd Meeting</h3>
title = soup.find("h3", attrs={"class": "BCTituloPagina ng-binding"})
print(title)

然而，命令

print(soup)

既没有返回标题：第123次会议也没有返回正文：根据....目标25个基点。

Answer 1

您无法使用python中的常规请求库来提取标题，因为您尝试提取的元素是使用javascript呈现的。您需要使用硒来实现目标。

代码：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()
driver.get('https://www.bcb.gov.br/en/#!/c/copomstatements/1724')
WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, '//h3')))
title = driver.find_element_by_xpath('//h3').text
print(title)
driver.close()

输出：

123rd Meeting

无法使用BeautifulSoup4抓取网站

1 个答案: