Question

I'm scraping a wikipedia page to extract all image urls and here's the code for it.

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule

class WikiSpider(CrawlSpider):
    name = 'wiki'
    allowed_domains = ['en.wikipedia.org']
    start_urls = ['https://en.wikipedia.org/wiki/Katy_Perry']

    rules = [Rule(LinkExtractor(restrict_xpaths=('//a[@class="image"]')), 
             callback='parse_item', follow=False),] 

    def parse_item(self, response):
        print(response.url)

When I run the spider, it isn't showing any results, but when I change the xpath inside restrict_xpaths it prints some random links. I need hrefs in the xpath '//a[@class="image"]' but it isn't working, what's the reason ? I know that I could use basic spider instead of CrawlSpider and avoid rules altogether. But I wanna know why the xpath I entered didn't work and what kind of xpaths and html tags does restrict_xpaths accept ?

Answer 1

您想要的链接是图片：

$ scrapy shell "https://en.wikipedia.org/wiki/Katy_Perry" -s USER_AGENT='Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.82 Safari/537.36'
2016-08-19 11:17:05 [scrapy] INFO: Scrapy 1.1.2 started (bot: scrapybot)
(...)
2016-08-19 11:17:06 [scrapy] DEBUG: Crawled (200) <GET https://en.wikipedia.org/wiki/Katy_Perry> (referer: None)
(...)
In [1]: response.xpath('//a[@class="image"]/@href').extract()
Out[1]: 
['/wiki/File:Katy_Perry_DNC_July_2016_(cropped).jpg',
 '/wiki/File:Katy_Perry_performing.jpg',
 '/wiki/File:Katy_Perry%E2%80%93Zenith_Paris.jpg',
 '/wiki/File:PWT_Cropped.jpg',
 '/wiki/File:Alanis_Morissette_5-19-2014.jpg',
 '/wiki/File:Freddie_Mercury_performing_in_New_Haven,_CT,_November_1977.jpg',
 '/wiki/File:Katy_Perry_California_Dreams_Tour_01.jpg',
 '/wiki/File:Katy_Perry_UNICEF_2012.jpg',
 '/wiki/File:Katy_Perry_Hillary_Clinton,_I%27m_With_Her_Concert.jpg',
 '/wiki/File:Wikiquote-logo.svg',
 '/wiki/File:Commons-logo.svg']

默认链接提取器过滤器a lot of extensions，包括图像：

In [2]: from scrapy.linkextractors import LinkExtractor

In [3]: LinkExtractor(restrict_xpaths=('//a[@class="image"]')).extract_links(response)
Out[3]: []

您可以use deny_extensions=[]不过滤任何内容：

In [4]: LinkExtractor(restrict_xpaths=('//a[@class="image"]'), deny_extensions=[]).extract_links(response)
Out[4]: 
[Link(url='https://en.wikipedia.org/wiki/File:Katy_Perry_DNC_July_2016_(cropped).jpg', text='', fragment='', nofollow=False),
 Link(url='https://en.wikipedia.org/wiki/File:Katy_Perry_performing.jpg', text='', fragment='', nofollow=False),
 Link(url='https://en.wikipedia.org/wiki/File:Katy_Perry%E2%80%93Zenith_Paris.jpg', text='', fragment='', nofollow=False),
 Link(url='https://en.wikipedia.org/wiki/File:PWT_Cropped.jpg', text='', fragment='', nofollow=False),
 Link(url='https://en.wikipedia.org/wiki/File:Alanis_Morissette_5-19-2014.jpg', text='', fragment='', nofollow=False),
 Link(url='https://en.wikipedia.org/wiki/File:Freddie_Mercury_performing_in_New_Haven,_CT,_November_1977.jpg', text='', fragment='', nofollow=False),
 Link(url='https://en.wikipedia.org/wiki/File:Katy_Perry_California_Dreams_Tour_01.jpg', text='', fragment='', nofollow=False),
 Link(url='https://en.wikipedia.org/wiki/File:Katy_Perry_UNICEF_2012.jpg', text='', fragment='', nofollow=False),
 Link(url="https://en.wikipedia.org/wiki/File:Katy_Perry_Hillary_Clinton,_I'm_With_Her_Concert.jpg", text='', fragment='', nofollow=False),
 Link(url='https://en.wikipedia.org/wiki/File:Wikiquote-logo.svg', text='', fragment='', nofollow=False),
 Link(url='https://en.wikipedia.org/wiki/File:Commons-logo.svg', text='', fragment='', nofollow=False)]

为什么restrict_xpath忽略了<a> tags?

1 个答案: