Question

请帮我优化我的scrapy蜘蛛。特别是下一页分页不起作用。每页有很多页面有50个项目。我在parse_items中捕获第一页50项（链接），并在parse_items中删除了下一页项目。

import scrapy
from scrapy import Field
from fake_useragent import UserAgent 

class DiscoItem(scrapy.Item):
    release = Field()
    images = Field()


class discoSpider(scrapy.Spider):
    name = 'myspider'
    allowed_domains = ['discogs.com']
    query = input('ENTER SEARCH MUSIC TYPE : ')
    start_urls =['http://www.discogs.com/search?q=%s&type=release'%query]
   custome_settings = {
'User-Agent': "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.186 Safari/537.36",

   'handle_httpstatus_list' : [301,302,],
    'download_delay' :10}

    def start_requests(self):
        yield scrapy.Request(url=self.start_urls[0], callback=self.parse)   

    def parse(self, response):
        print('START parse \n')
        print("*****",response.url)

        #next page pagination 
        next_page =response.css('a.pagination_next::attr(href)').extract_first()
       next_page = response.urljoin(next_page)
       yield scrapy.Request(url=next_page, callback=self.parse_items2)

        headers={}
        for link in response.css('a.search_result_title ::attr(href)').extract():

        ua = UserAgent()# random user agent
        headers['User-Agent'] = ua.random
        yield scrapy.Request(response.urljoin(link),headers=headers,callback=self.parse_items) 


    def parse_items2(self, response):
        print('parse_items2 *******', response.url)
        yield scrapy.Request(url=response.url, callback=self.parse)  

    def parse_items(self,response):

        print("parse_items**********",response.url)
        items = DiscoItem()
        for imge in response.css('div#page_content'):
            img = imge.css("span.thumbnail_center img::attr(src)").extract()[0]
            items['images'] = img
            release=imge.css('div.content a ::text').extract()
            items['release']=release[4]
            yield items

Answer 1

当我尝试运行你的代码时（在修复了许多缩进，拼写和字母大小写错误之后），这行显示在scrapy的日志中：

2018-03-05 00:47:28 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET https://www.discogs.com/search/?q=rock&type=release&page=2> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)

默认情况下，Scrapy会过滤重复的请求，而您的parse_items2()方法除了创建重复请求外什么也不做。我没有看到任何理由存在该方法。

你应该做的是将˙parse（）`方法指定为你的请求的回调，并避免使用一个不做任何事情的额外方法：

yield scrapy.Request(url=next_page, callback=self.parse)

Answer 2

尝试进行分页：

try:
    nextpage = response.urljoin( response.xpath("//*[contains(@rel,'next') and contains(@id,'next')]/@url")[0].extract() )
    yield scrapy.Request( nextpage, callback=self.parse )
except:
    pass

Scrapy分页不起作用并优化蜘蛛

2 个答案: