Question

我正在尝试爬网此页面以及下一页的链接。 https://www.chavesnamao.com.br/carros/usados/brasil/

<a href="javascript:;" onclick="goToPage(2); return false;" rel="next" title="Página 2">2</a>

使用此goToPage()函数生成指向下一页的链接。该函数生成这样的查询字符串；

{"pg":"1","or":"","c":"","e":"","r":"50","view":"tabela"}

pg是请求的页面参数。是否有任何系统的方法来实现在这些页面之后的Rule并调用解析函数，还是我需要对增加pg参数并每次发送请求的循环进行硬编码？这是我的问题，下面是我的蜘蛛。

class MySpider(CrawlSpider):

    name = 'myspider'

    start_urls = [
        'https://www.chavesnamao.com.br/carros/usados/brasil/#{%22pg%22:%221%22,%22or%22:%22%22,%22c%22:%22%22,%22e%22:%22%22,%22r%22:%2250%22,%22view%22:%22tabela%22}',
    ]

    def parse(self, response):

        #logging.warning('parse function called on %s', response.url)

        #from scrapy.shell import inspect_response
        #inspect_response(response, self)

        items = response.css('div.view ul#listagem li div a.description::attr(href)').getall()

        for item in items:
            yield scrapy.Request(
                url='{}'.format(item),
                method='GET',
                callback=self.parse_items,
                headers={'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'}
            )

    def parse_items(self, response):

        # Crawling the item without any problem here

Scrapy-提取使用JS函数生成的页面链接

0 个答案: