Question

我试图在next.co.uk中取消男士大衣和夹克的类别，但我意识到该页面具有无限滚动页面

# -*- coding: utf-8 -*-
import scrapy
from ..items import NextItem

class NewoneSpider(scrapy.Spider):
    name = 'newOne'
    allowed_domains = ['www.next.co.uk']
    start_urls = [
        'https://www.next.co.uk/shop/gender-newbornboys-gender-newbornunisex-gender-olderboys-gender-youngerboys-productaffiliation-coatsandjackets-0'
        ]

    def parse(self, response):
        items = NextItem();
        global productCategory
        global productSubCategory
        products = response.css('.Details')
        currentUrl = response.request.url

        for product in products:
            productCategory = 'Furniture'
            productSubCategory = 'living Room'
            productCountry = 'uk'
            productSeller = 'John Lewis'
            productLink = product.css('.TitleText::attr(href)').extract_first()
            productTitle = product.css('.Desc::text').extract_first()
            productImage = product.css('.Image img::attr(src)').extract_first()
            productSalePrice = product.css('.Price a::text').extract_first()

            items['productCategory'] = productCategory
            items['productSubCategory'] = productSubCategory
            items['productCountry'] = productCountry
            items['productSeller'] = productSeller
            items['productLink'] = productLink
            items['productTitle'] = productTitle
            items['productImage'] = productImage
            items['productSalePrice'] = productSalePrice

            yield items

我能够抓取28个项目，并且在具有无限滚动实现功能的网站上可以看到更多的内容。

Answer 1

当您向下滚动页面时，会将XHR呼叫发送到服务器，并请求更多数据。示例：

https://www.next.co.uk/shop/gender-newbornboys-gender-newbornunisex-gender-olderboys-gender-youngerboys-productaffiliation-coatsandjackets/isort-score-minprice-0-maxprice-30000-srt-24

每个请求几乎都是相同的，但url中的最后一个元素将增加24：

srt-24
srt-48
srt-72

现在您知道“无穷大”的工作原理，可以尝试使用代码对其进行仿真。

示例：

import requests

URL_TEMPLATE = 'https://www.next.co.uk/shop/gender-newbornboys-gender-newbornunisex-gender-olderboys-gender-youngerboys-productaffiliation-coatsandjackets/isort-score-minprice-0-maxprice-30000-srt-{}'

for step in range(24, 240, 24):
    r = requests.get(URL_TEMPLATE.format(step))
    if r.status_code == 200:
        # TODO We have the data - lets parse it
        pass

Answer 2

如果我有两个链接想要从中刮除 https://www.next.co.uk/shop/gender-newbornboys-gender-newbornunisex-gender-olderboys-gender-youngerboys-productaffiliation-coatsandjackets-0 和 https://www2.next.co.uk/shop/gender-men-productaffiliation-coatsandjackets-0 代码怎么样

如何抓取和无限滚动页面？

2 个答案: