Question

我想要做的很简单，转到https://www.reddit.com/new/，然后仅提取前3个帖子的标题。在尝试进行下一个2之前，我尝试仅提取第一个的标题，但是我一直遇到问题。我将不胜感激。

import urllib
from bs4 import BeautifulSoup
import requests


quote_page = 'https://www.reddit.com/r/new/'
page = urllib.urlopen(quote_page)
soup = BeautifulSoup(requests.get(quote_page).text, 'html.parser')
title_box = soup.find('div', {'class':'top-matter'})

title = title_box.text.strip()
print(title)

错误输出：

Traceback (most recent call last):
  File "/home/ad044/Desktop/sidebar stuff/123.py", line 13, in <module>
    title = title_box.text.strip()
AttributeError: 'NoneType' object has no attribute 'text'
[Finished in 1.8s with exit code 1]
[shell_cmd: python -u "/home/ad044/Desktop/sidebar stuff/123.py"]
[dir: /home/ad044/Desktop/sidebar stuff]
[path: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin]

Answer 1

页面使用javascript，因此您需要一种类似硒的方法，该方法可以渲染您感兴趣的元素。然后您可以索引到列表

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://www.reddit.com/new/'
driver = webdriver.Chrome()
driver.get(url)
data = WebDriverWait(driver,10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, ".kCqBrs")))[:3]
for item in data:
    print(item.text)

从“ div”中的“ p”中提取文本

1 个答案: