Scrapy结果正在重复

Question

我正在尝试使用链接提取器从此网站https://pagalworld.me/category/11598/Latest%20Bollywood%20Hindi%20Mp3%20Songs%20-%202017.html获取歌曲的名称，但结果正在重复。

import scrapy
from scrapy import Request
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
class RedditSpider(CrawlSpider):
    name='pagalworld'
    allowed_domains = ["pagalworld.me"]
    start_urls=['https://pagalworld.me/category/11598/Latest%20Bollywood%20Hindi%20Mp3%20Songs%20-%202017.html']
    rules = ( 
        Rule(
        LinkExtractor(restrict_xpaths='//div/ul'), 
        follow=True, 
        callback='parse_start_url'),
    )  
    def parse_start_url(self, response):
        songName= response.xpath('//li/b/a/text()').extract()

        for item in songName:

            yield {"songName":item,
        "URL":resposne}  

Answer 1

你的蜘蛛似乎一切都是正确的。但是，如果您查看歌曲页面，它会为每首歌曲提供两个版本：

$ scrapy shell "https://pagalworld.me/files/12450/Babumoshai%20Bandookbaaz%20(2017)%20Movie%20Mp3%20Songs.html"
>[1]: response.xpath('//li/b/a/text()').extract()
<[1]: 
['03 Aye Saiyan - Babumoshai Bandookbaaz 190Kbps.mp3',
 '03 Aye Saiyan - Babumoshai Bandookbaaz 320Kbps.mp3',
 '01 Barfani - Male (Armaan Malik) 190Kbps.mp3',
 '01 Barfani - Male (Armaan Malik) 320Kbps.mp3',
 '02 Barfani - Female (Orunima Bhattacharya) 190Kbps.mp3',
 '02 Barfani - Female (Orunima Bhattacharya) 320Kbps.mp3']

一个版本的质量低于190kbps，另一个版本的质量高于320kbps 在这里你可能只想保留其中一个：

>[2]: response.xpath('//li/b/a/text()[contains(.,"320Kb")]').extract()
<[2]: 
['03 Aye Saiyan - Babumoshai Bandookbaaz 320Kbps.mp3',
 '01 Barfani - Male (Armaan Malik) 320Kbps.mp3',
 '02 Barfani - Female (Orunima Bhattacharya) 320Kbps.mp3']

修改： 好像还有重复问题。尝试在链接提取器上禁用follow=True，因为在这种情况下您不想关注。