无法使用漂亮的汤提取网页数据

Question

  url =  "https://www.telegraph.co.uk/formula-1/2018/08/25/f1-live-belgian-grand-prix-2018-qualifying-latest-updates/"
  soup = bs(urlopen(url), "lxml")
  divs = soup.findAll('div')
  base_url = "https://www.telegraph.co.uk"
  images = []
  print (divs)
  []

我得到空输出。我认为该页面充满活力。如何从此页面提取div。

Answer 1

页面内容是由JS /动态加载的，因此您必须使用硒...您可以执行以下操作...

from bs4 import BeautifulSoup
from selenium import webdriver#you need to install selenium
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
#copy your chromedriver to python folder
driver = webdriver.Chrome(chrome_options=options)
url =  ("https://www.telegraph.co.uk/"
"formula-1/2018/08/25/f1-live-belgian"
"-grand-prix-2018-qualifying-latest-updates/")
driver.get(url)
soup = BeautifulSoup(driver.page_source, 'lxml')
divs = soup.findAll('div')
print(divs)