Question

我正在尝试制作网络抓取工具，但无法获取下一页的链接。我尝试了一些组合，但没有一个起作用。 scrapy.org上的教程格式更简单，因此无法解决我的问题

我要抓取的网站具有以下布局：

<nav class="nav_class">
    <a class="class_1" href="1.html">
    <a class="class_2" href="2.html">
    <a class="class_3" href="3.html">

我想使用CSS选择器获取3.html链接

import scrapy


class MySpider(scrapy.Spider):
    name = "flip_spider"

    def start_requests(self):
        urls = [
            "https://www.flipkart.com/mobiles/pr?sid=tyy%2C4io&p%5B%5D=facets.processor_brand%255B%255D%3DSnapdragon&p%5B%5D=facets.serviceability%5B%5D%3Dfalse&p%5B%5D=facets.offer_type%255B%255D%3DExchange%2BOffer&otracker=clp_banner_1_10.bannerX3.BANNER_mobile-phones-store_HPUGCU9BYBF6&fm=neo%2Fmerchandising&iid=M_934db066-154e-4074-a4b1-96f56a0af28e_6.HPUGCU9BYBF6&ppt=HomePage&ppn=Home&ssid=85m4yqvgzk0000001558978084715&page=1",
        ]

        for url in urls:
            yield scrapy.Request(url, callback=self.parse)

    def parse(self, response):
        # page_id=response.url.split("=")[-1]
        phone_details = response.css("div._1-2Iqu.row")

        for ph in phone_details:
            phone = ph.css("div._3wU53n::text").get()
            rating = ph.css("div.hGSR34::text").get()
            price = ph.css("div._1vC4OE._2rQ-NK::text").get()

            yield{
                "name": phone,
                "rating": rating,
                "price": price,
            }
        final = "https://www.flipkart.com/mobiles/pr?sid=tyy%2C4io&p%5B%5D=facets.processor_brand%255B%255D%3DSnapdragon&p%5B%5D=facets.serviceability%5B%5D%3Dfalse&p%5B%5D=facets.offer_type%255B%255D%3DExchange%2BOffer&otracker=clp_banner_1_10.bannerX3.BANNER_mobile-phones-store_HPUGCU9BYBF6&fm=neo%2Fmerchandising&iid=M_934db066-154e-4074-a4b1-96f56a0af28e_6.HPUGCU9BYBF6&ppt=HomePage&ppn=Home&ssid=85m4yqvgzk0000001558978084715&page=6"
        next_page_id = response.css("nav._1ypTlJ a._3fVaIS::attr(href)").get()

        # ^This is the line I need help with

        if next_page_id is not final:
            next_page = response.urljoin(next_page_id)
            yield scrapy.Request(next_page, callback=self.parse)

它只会抓取第一页然后停止

Answer 1

将您的代码更改为此，它将起作用

next_page_id = response.css("nav._1ypTlJ a._3fVaIS::attr(href)").get()

if next_page_id:
    next_page = response.urljoin(next_page_id)
    yield scrapy.Request(next_page, callback=self.parse)

使用scrapy进入下一页

1 个答案: