Question

我有强迫scrapy转到另一页的问题。我试图获得不同月份的所有Opera时间表。

我需要的每个地址都是这样的：“”https://www.mail-archive.com/user@cassandra.apache.org/ +月份名称

这就是为什么我已经列出了几个月并试图迭代它们，但不知何故Scrapy只是忽略了它。我试图打印“next_page”收集的所有网址，它们都是正确的。

import scrapy
from ..items import ShowItem, ShowItemLoader
from scrapy.selector import HtmlXPathSelector


class OperaSpider(scrapy.Spider):
    name = "opera"
    allowed_domains = ["http://www.opera.krakow.pl"]
    start_urls = [
        "http://www.opera.krakow.pl/pl/repertuar/na-afiszu/listopad"

]
    shows_list_xpath = '//div[@class="row-fluid row-performance    "]'
    item_fields = {
        'month':'.//ul[@class="nav nav-pills nav-repertuar"]/li[@class="active"]/a/text()',
        'title': './/h2[@class="item-title"]/a/text()',
        'time': './/div[@class="item-time vertical-center"]/div[@class="vcentered"]/text()',
        'date': './/div[@class="item-date vertical-center"]/div[@class="vcentered"]/text()',

}


def parse(self, response):

    selector = HtmlXPathSelector(response)

    for show in selector.select(self.shows_list_xpath):
        loader = ShowItemLoader(ShowItem(), selector=show)

        for field, xpath in self.item_fields.iteritems():
            loader.add_xpath(field, xpath)
        yield loader.load_item()


    list = ["styczen", "luty"
        , "marzec", "kwiecien"
        , "maj", "czerwiec"
        , "lipiec", "sierpien"
        , "wrzesien", "pazdziernik"
        , "listopad", "grudzien"]

    for i in list:
        next_page = ("http://www.opera.krakow.pl/pl/repertuar/na-afiszu/%s" % i)
        yield scrapy.Request(next_page, callback=self.parse)

Answer 1

scrapy只检查allowed_domains请求网址的netloc，您需要将http://www.opera.krakow.pl更改为opera.krakow.pl。

Scrapy不想去下一个网址

1 个答案: