Question

我正在尝试从网页上抓取数据，但无法找到容器内的所有数据块。

<div class="product-list-container" data-project-b="" data-mini-basket="on" data-chunk-caching-gutter="5" data-chunk-caching-delay="2" data-total-results="112" data-chunks-per-page="8" data-results-per-chunk="24" data-current-chunk="1" data-page-end-chunk="5" data-offset-chunk-threshold="12" data-findability-grid-position="on">

在上述容器中，我有五个数据块，如下例所示：

<div class="product-list-container__chunk product-list-container__chunk--1" data-chunk="1" data-tagg-processed="true">

<div class="product-list-container__chunk product-list-container__chunk--2" data-chunk="2" style="" data-tagg-processed="true">

我可以通过以下方式访问第一个容器：

from bs4 import BeautifulSoup
import requests

soup = BeautifulSoup(page.content, 'html.parser')
d = soup.find('div', attrs={'product-list-container__chunk product-list-container__chunk--1'})

但是，如果我尝试对第二个容器执行相同操作，则无法访问它，似乎解析器无法从网站上获取所有信息，或者我可能没有正确寻找它。

from bs4 import BeautifulSoup
import requests

soup = BeautifulSoup(page.content, 'html.parser')
d = soup.find('div', attrs={'product-list-container__chunk product-list-container__chunk--2'})

Answer 1

页面随着向下滚动而动态加载产品。您可以使用此脚本获取有关所有产品的信息：

import requests
from bs4 import BeautifulSoup


url = 'https://www.johnlewis.com/browse/electricals/mobile-phones-accessories/view-all-mobile-phones/_/N-a8v?incremental=true&page={page}&country=uk&currency=gbp'

page = 1
while True:
    soup = BeautifulSoup(requests.get(url.format(page=page)).content, 'html.parser')

    titles = soup.select('.product-card__title')

    if not titles:
        break

    # print some information on screen:
    for t in titles:
        print(t.get_text(strip=True))
        print(t.find_next(class_='product-card__price').get_text(strip=True))
        print('-' * 80)

    page += 1

打印：

Apple iPhone 11, iOS, 6.1", 4G LTE, SIM Free, 64GB
£679.00
--------------------------------------------------------------------------------
Apple iPhone 11 Pro Max, iOS, 6.5", 4G LTE, SIM Free, 256GB
£1,174.00
--------------------------------------------------------------------------------
Apple iPhone 11 Pro, iOS, 5.8", 4G LTE, SIM Free, 256GB
£1,074.00
--------------------------------------------------------------------------------
Apple iPhone 11 Pro Max, iOS, 6.5", 4G LTE, SIM Free, 64GB
£1,024.00
--------------------------------------------------------------------------------

...and so on.

找不到Python容器中的网页抓取

1 个答案: