Question

我正在尝试从NYT Cooking的食谱中删除评分，但在获取所需的内容时遇到了问题。当我在NYT页面上查看源代码时，会看到以下内容：

<div class="ratings-rating">
    <span class="ratings-header ratings-content">194 ratings</span>

    <div class="ratings-stars-wrap">
      <div class="ratings-stars ratings-content four-star-rating avg-rating">

我要提取的内容是194 ratings和four-star-rating。但是，当我通过Beautiful Soup插入页面源时，只会看到以下内容：

 <div class="ratings-rating">
    <span class="ratings-header ratings-content"><%= header %></span>

    <div class="ratings-stars-wrap">
      <div class="ratings-stars ratings-content <%= ratingClass %> <%= state %>">

我正在使用的代码是：

url = 'https://cooking.nytimes.com/recipes/1020049-lemony-chicken-soup-with-fennel-and-dill'

r = get(url, headers = headers, timeout=15)
page_soup = soup(r.text,'html.parser')

有人在想为什么这些信息不能通过？

Answer 1

尝试使用以下代码

import requests
import lxml
from lxml import html
import re

url = "https://cooking.nytimes.com/recipes/1019706-spiced-roasted-cauliflower-with-feta-and-garlic?action=click&module=Recirculation%20Band%20Recipe%20Card&region=More%20recipes%20from%20Alison%20Roman&pgType=recipedetails&rank=1"

r = requests.get(url)
tree = html.fromstring(r.content)

t = tree.xpath('/html/body/script[14]')[0]

# look for value for bootstrap.recipe.avg_rating
m = re.search("bootstrap.recipe.avg_rating = ", t.text)
colon = re.search(";", t.text[m.end()::])
rating = t.text[m.end():m.end()+colon.start()]
print(rating)

# look for value for bootstrap.recipe.num_ratings = 
n = re.search("bootstrap.recipe.num_ratings = ", t.text)
colon2 = re.search(";", t.text[n.end()::])
star = t.text[n.end():n.end()+colon2.start()]
print(star)

Answer 2

使用属性=值选择器更容易从类ratings-metadata的范围中抓取

import requests
from bs4 import BeautifulSoup

data = requests.get('https://cooking.nytimes.com/recipes/1020049-lemony-chicken-soup-with-fennel-and-dill')
soup = BeautifulSoup(data.content, 'lxml')
rating = soup.select_one('[itemprop=ratingValue]').text
ratingCount = soup.select_one('[itemprop=ratingCount]').text
print(rating, ratingCount)

在使用Beautiful Soup抓取网站时无法加载某些内容

2 个答案: