Question

这仅会刮掉4个链接，并返回15个刮擦项/行，但我需要用35个以上刮擦项刮擦20个链接。

如果有人帮助我，那对我会很有帮助

sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssp

from scrapy import Spider
from scrapy.http import Request


class TastySpider(Spider):
    name = 'tasty'
    allowed_domains = ['tasty.co']
    start_urls = ['https://tasty.co/topic/game-day']


    def parse(self, response):
        lists=response.css('a.feed-item::attr(href)').extract()
        for link in lists:
            link=response.urljoin(link)
            if link:
                yield Request(link, callback=self.parse_page)


    def parse_page(self, response):
        Recipe_URL=response.url
        Title=response.xpath('//h1//text()').extract_first()
        instruction=response.css('ol.prep-steps li.xs-mb2::text').extract()
        Cooking_time=response.xpath('//*[@class="recipe-time-container xs-flex xs-mx2 xs-relative xs-b0 md-b2 xs-mt2 md-mt0"]//p//text()').extract_first()
        Image_video_URL=response.xpath('//*[@property="og:image"]//@content').extract_first()
        servings=response.xpath('//h2[contains(text(),"Ingredients")]//following-sibling::p//text()').extract_first()

        Us_measurement=response.xpath('//*[@class="measurement us-measurement xs-inline-block"]//text()').extract()

        Ingredient=response.xpath('//*[@class="xs-mb1 xs-mt0"]//section//following-sibling::text()').extract()

        Non_us_measurement=response.xpath('//*[@class="measurement non-us-measurement xs-inline-block"]//text()').extract()


        for Us_measur, Ingred, Non_us in zip(Us_measurement, Ingredient, Non_us_measurement):
            Us_measur=Us_measur.strip()

            Ingred=Ingred.strip()

            Non_us=Non_us.strip()


            yield {
                    'Us-measurement':Us_measur,
                    'Ingredient':Ingred,
                    'Non-us-measurement':Non_us,
                    'Recipe_URL':Recipe_URL,
                    'Title':Title,
                    'instruction':instruction,
                    'Cooking_time':Cooking_time,
                    'Servings':servings,
                    'Image-video_URL':Image_video_URL


            }

Answer 1

Scrapy实际上会抓取您希望抓取的20个链接。

但是，只有/recipe/链接包含您要抓取的数据。
/compilation/链接仅包含其他食谱的链接。

解决此问题的最简单方法可能是使用CrawlSpider，并为配方和编译链接创建单独的规则。

Scrapy仅抓取了4页

1 个答案: