单击Scrapy + Splash的按钮以加载项目

时间:2019-09-26 21:31:58

标签: python web-scraping

我正在抓取此网址https://www.dentalcremer.com.br/departamento/854656/adesivo,我想单击最后的按钮以加载更多项目。试图查看互联网,但未找到有效的解决方案 脚本:

import scrapy
from scrapy.loader import ItemLoader
from ..items import DenItem
from scrapy_splash import SplashRequest


class DenSpider(scrapy.Spider):
    name = 'den'

    def start_requests(self):
        url = input("Enter URL: ")
        yield SplashRequest(
            url=url,

            callback=self.parse)

    def parse(self, response):
        for quote in response.selector.css('.collection-product'):
            loader = ItemLoader(item=DenItem(), selector=quote, response=response)
            loader.add_css('product_name', '.collection-product-name a::text')
            loader.add_css('t_new_price', '.collection-product-price a::text')
            loader.add_css('u_desc', '.collection-product-description a::text')
            loader.add_css('v_old_price', 'del::text')

            yield loader.load_item()

0 个答案:

没有答案