Question

我必须从产品页面获取所有评论文字和分数，我设法做到了：

通过在具有单个产品评论的页面上添加手动链接，我可以从页面（包括评论的其他页面）获得所有评论和分数

为加快此过程，我想从类别页面转到产品页面，并在完成此操作后获取所有评论和评分，然后继续开发另一种产品。

import scrapy


class ReviewAutoSpider(scrapy.Spider):
    name = 'automatic'

    start_urls = ['https://www.ceneo.pl/Gry_bez_pradu']

    def parse(self, response):
        # follow links to website with review
        for href in response.css('a.product-rewiews-link + a::attr(href)'):
            yield response.follow(href, self.parse_link)

        # follow pagination links
        #for href in response.css('li.arrow-next a::attr(href)'):
         #   yield response.follow(href, self.parse)

    def parse_link(self, response):
        #get all reviews+score on page 
        for review in response.css('li.review-box'):
            yield {
             'score': review.css('span.review-score-count::text').get(),
             'text': review.css('p.product-review-body::text').getall(),
            }
        # follow pagination links
        for href in response.css('li.arrow-next a::attr(href)'):
            yield response.follow(href, callback=self.parse)

Answer 1

好的，下面的解决方案应该起作用。您仅获得的链接具有链接的第二部分'/ 19838632'，您需要使用response.urljoin('/19838632')来获取完整的链接。另外，当前设置Spider的方式将使您同时向站点发出大量请求，因此，我强烈建议您使用代理服务。

`python

import scrapy
class ReviewAutoSpider(scrapy.Spider):

    name = 'automatic'

    start_urls = ['https://www.ceneo.pl/Gry_bez_pradu']

    def parse(self, response):
        # follow links to website with review
        for href in response.css('a.product-rewiews-link + a::attr(href)'):
            yield scrapy.Request(href, callback = self.parse)

        for href in response.css('.cat-prod-row-name a::attr(href)').extract():
            link = response.urljoin(href)
            yield scrapy.Request(link, callback = self.parse)

        next_page_link = response.css('li[class ="page-arrow arrow-next"] a::attr(href)').extract_first()
        next_page_link = response.urljoin(next_page_link)
        yield scrapy.Request(next_page_link, callback = self.parse)


    def parse_link(self, response):
        #get all reviews+score on page
        for review in response.css('li.review-box'):
            yield {
             'score': review.css('span.review-score-count::text').get(),
             'text': review.css('p.product-review-body::text').getall(),
            }
        # follow pagination links
        for href in response.css('li.arrow-next a::attr(href)'):
            yield scrapy.Request(href, callback = self.parse)

`

如何通过链接到我需要的ifnormation进行爬网

1 个答案: