Question

尝试刮取snapdeal数据时的输出如下：

scrapy shell "https://www.snapdeal.com"

response.text

u'<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don\'t have permission to access "http&#58;&#47;&#47;www&#46;snapdeal&#46;com&#47;" on this server.<P>\nReference&#32;&#35;18&#46;1dd70b17&#46;1514632273&#46;17456300\n</BODY>\n</HTML>\n'

任何帮助？

Answer 1

如果我使用User-Agent，那么我会得到正确的页面

scrapy shell

fetch("https://www.snapdeal.com", headers={'User-Agent': "Mozilla/5.0"})

response.text

或使用脚本

import scrapy
#from scrapy.commands.view import open_in_browser

class MySpider(scrapy.Spider):

    name = 'myspider'

    start_urls = ['https://www.snapdeal.com/']

    def parse(self, response):
        print('url:', response.url)

        #open_in_browser(response)

        for item in response.xpath('//*[@class="catText"]/text()').extract():
            print(item)

# --- it runs without project ---

from scrapy.crawler import CrawlerProcess

c = CrawlerProcess({
    'USER_AGENT': 'Mozilla/5.0',
})
c.crawl(MySpider)
c.start()

无法使用scrapy刮取snapdeal数据

1 个答案: