Question

我正试图从一个名为WG-Gesucht的德国网站上提取公寓/房间的信息。我有点想到他们的链接遵循逻辑：

http:// www.wg-gesucht.de/wohnungen-in-Berlin.8.2.0.**X**.html`

其中X=0, 1, 2, ...

当我将链接粘贴到浏览器中时，它们确实可以正常工作。然而，当我尝试抓取这些链接时，我的乐观情绪破灭了。最后，我只在我的数据库中获得与X = 0相对应的条目。

这是我的蜘蛛：

from scrapy.http.request import Request
from scrapy.spider import Spider
from scrapy.selector import HtmlXPathSelector
from scrapy.contrib.loader import XPathItemLoader
from scrapy.contrib.loader.processor import Join, MapCompose

from scraper_app.items import WGGesuchtEntry




class WGGesuchtSpider(Spider):
    """Spider for wg-gesucht.de, Berlin"""
    name = "wggesucht"
    allowed_domains = ["wg-gesucht.de"]
    start_urls = ["http://www.wg-gesucht.de/wohnungen-in-Berlin.8.2.0.0.html"]
    # start_urls = ["http://www.wg-gesucht.de/wohnungen-in-Berlin.8.2.0.%s.html"%x for x in range(0,1)]


    entries_list_xpath = '//tr[contains(@id,"ad--")]'
    item_fields = {
        # 'title': './/span[@itemscope]/meta[@itemprop="name"]/@content',
        'rooms': './/td[2]/a/span/text()',
        'entry_date': './/td[3]/a/span/text()',
        'price': './/td[4]/a/span/b/text()',
        'size': './/td[5]/a/span/text()',
        'district': './/td[6]/a/span/text()',
        'start_date': './/td[7]/a/span/text()',
        'end_date': './/td[8]/a/span/text()',
        'link': './/@adid'
    }

    def start_requests(self):
        for i in xrange(1, 10):
            url = 'http://www.wg-gesucht.de/wohnungen-in-Berlin.8.2.0.' + str(i) + '.html'
            yield Request(url=url, callback=self.parse_items)


    def parse_items(self, response):
        """
        Default callback used by Scrapy to process downloaded responses

        # Testing contracts:
        # @url http://www.livingsocial.com/cities/15-san-francisco
        # @returns items 1
        # @scrapes title link

        """
        selector = HtmlXPathSelector(response)

        # iterate over deals
        for entry in selector.xpath(self.entries_list_xpath):
            loader = XPathItemLoader(WGGesuchtEntry(), selector=entry)

            # define processors
            loader.default_input_processor = MapCompose(unicode.strip)
            loader.default_output_processor = Join()

            # iterate over fields and add xpaths to the loader
            for field, xpath in self.item_fields.iteritems():
                loader.add_xpath(field, xpath)
            yield loader.load_item()

我应该使用CrawlSpider代替Spider吗？

欢迎任何建议，谢谢！

Answer 1

看起来像Cookie问题，您可以通过打开新浏览器并直接尝试6th page来检查，例如，您将收到第一个页面的响应。< / p>

Scrapy尝试将Cookie用于后续请求，因此解决此问题的一种方法是不将请求迭代到页面，而是一个接一个地进行：

import re

start_urls = [http://example.com/0.html]

def parse(self, response):
    cur_index = response.meta.get('cur_index', 1)
    ...
    new_url = # use the response.url to change to the following url (+1 to the index)
    if cur_index < 10:
        yield Request(new_url, callback=self.parse, meta={'cur_index': cur_index+1})

Scrapy一遍又一遍地抓住同一页面为德国网站上的不同网址

1 个答案: