无法使用scrapy刮取snapdeal数据

时间:2017-12-30 11:16:34

标签: python html web-scraping scrapy

尝试刮取snapdeal数据时的输出如下:

scrapy shell "https://www.snapdeal.com"

response.text

u'<HTML><HEAD>\n<TITLE>Access Denied</TITLE>\n</HEAD><BODY>\n<H1>Access Denied</H1>\n \nYou don\'t have permission to access "http&#58;&#47;&#47;www&#46;snapdeal&#46;com&#47;" on this server.<P>\nReference&#32;&#35;18&#46;1dd70b17&#46;1514632273&#46;17456300\n</BODY>\n</HTML>\n'

任何帮助?

1 个答案:

答案 0 :(得分:0)

如果我使用User-Agent,那么我会得到正确的页面

scrapy shell

fetch("https://www.snapdeal.com", headers={'User-Agent': "Mozilla/5.0"})

response.text

或使用脚本

import scrapy
#from scrapy.commands.view import open_in_browser

class MySpider(scrapy.Spider):

    name = 'myspider'

    start_urls = ['https://www.snapdeal.com/']

    def parse(self, response):
        print('url:', response.url)

        #open_in_browser(response)

        for item in response.xpath('//*[@class="catText"]/text()').extract():
            print(item)

# --- it runs without project ---

from scrapy.crawler import CrawlerProcess

c = CrawlerProcess({
    'USER_AGENT': 'Mozilla/5.0',
})
c.crawl(MySpider)
c.start()