如何从h2标签获取HREF？ Python /硒

Question

URL：https://www.nielsen.com/us/en/insights/related-tag/covid-19/

我有以下HTML元素：

我想从此类和标题文本中提取href。

我使用了driver.find_element_by_class_name('h2.entry-title h1>a').get_attribute('href)，但没有返回。

有什么建议吗？

Answer 1

首先您所做的是错误的。您可能想使用driver.find_element_by_css_selector()而不是driver.find_element_by_class_name()。但是，执行此操作的更好方法是使用请求库，因为您要查找的内容在页面源中可用。尝试运行以下脚本以使它们全部遍历多个页面。

import requests
from bs4 import BeautifulSoup

link = 'https://www.nielsen.com/us/en/insights/related-tag/covid-19/'

while True:
    r = requests.get(link)
    soup = BeautifulSoup(r.text,"html.parser")
    for item in soup.select("article"):
        article_link = item.select_one("h2.entry-title > a[href]")['href']
        article_title = item.select_one("h2.entry-title > a[href]").get_text(strip=True)
        print(article_link,article_title)

    try:
        link = soup.select_one("a[class^='next']")['href']
    except TypeError:
        break

如果您仍要坚持使用硒，可以按照以下方法进行操作：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

with webdriver.Chrome() as driver:
    driver.get("https://www.nielsen.com/us/en/insights/related-tag/covid-19/")

    while True:
        for elem in WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"article"))):
            article_link = WebDriverWait(elem,10).until(EC.presence_of_element_located((By.CSS_SELECTOR,"h2.entry-title > a[href]"))).get_attribute("href")
            article_title = WebDriverWait(elem,10).until(EC.presence_of_element_located((By.CSS_SELECTOR,"h2.entry-title > a[href]"))).text
            print(article_link,article_title)

        try:
            WebDriverWait(driver,5).until(EC.presence_of_element_located((By.CSS_SELECTOR, "a[class^='next']"))).click()
            WebDriverWait(driver,5).until(EC.staleness_of(elem))
        except:
            break