Question

我正在尝试抓取.aspx页面，但它会将我重定向到不存在的页面。为了解决这个问题，我尝试设置'dont_merge_cookies'：True和'dont_redirect'：True，并覆盖我的start_requests，但现在，它给了我一个错误“'Response'对象没有属性'body_as_unicode'”和我的响应类类型是'scrapy.http.response.Response'。

这是我的代码：

class Inon_Spider(BaseSpider):
    name = 'Inon'
    allowed_domains = ['www.shop.inonit.in']

    start_urls = ['http://www.shop.inonit.in/Products/Inonit-Men-Jackets/QUIRK-BOX/Toy-Factory-Jacket---Soldiers/pid-1177471.aspx?Rfs=&pgctl=713619&cid=CU00049295']

     #redirects to http://www.shop.inonit.in/Products/Inonit-Men-Jackets/QUIRK-BOX/Toy-Factory-Jacket---Soldiers/1177471

    def start_requests(self):
        start_urls = ['http://www.shop.inonit.in/Products/Inonit-Men-Jackets/QUIRK-BOX/Toy-Factory-Jacket---Soldiers/pid-1177471.aspx?Rfs=&pgctl=713619&cid=CU00049295']

        for i in start_urls:

            yield Request(i, meta = {
                     'dont_merge_cookies': True,
                     'dont_redirect': True,
                     'handle_httpstatus_list': [302]
                 },callback=self.parse)

    def parse(self, response):
        print "Response %s" %response.__class__

        resp = TextResponse
        item = DealspiderItem()
        hxs = HtmlXPathSelector(resp)

        title = hxs.select('//div[@class="aboutproduct"]/div[@class="container9"]/div[@class="ctl_aboutbrand"]/h1/text()').extract()
        price = hxs.select('//span[@id="ctl00_ContentPlaceHolder1_Price_ctl00_spnWebPrice"]/span[@class="offer"]/span[@id="ctl00_ContentPlaceHolder1_Price_ctl00_lblOfferPrice"]/text()').extract()
        prc = price[0].replace("Rs.  ","")

        description = []
        item['price'] = prc
        item['title'] = title
        item['description'] = description
        item['url'] = response.url
        return item

Scrapy：在抓取.aspx页面时遇到问题

0 个答案: