Question

我有一个简单的蜘蛛，可以爬行当地的itu告。在我尝试添加两个静态列之前，代码可以正常工作。我要做的就是添加提取信息的日期（拉项）和提取信息的状态（状态项）。这是一个自加载页面，因此当我添加提取日期时，我只会得到前10个结果（或仅是第一页）。如果仅添加状态，则只会得到两个结果。当我删除两者时，都会得到40多个结果。

我做了＃行不能正常工作：

Item.py文件：

import scrapy


class AlItem(scrapy.Item):
    name = scrapy.Field()
    link = scrapy.Field()
    obit = scrapy.Field()
    news = scrapy.Field()
    #pull = scrapy.Field()
    #state = scrapy.Field()

蜘蛛文件：

import scrapy
import time
from al.items import AlItem

class AlabamaSpider(scrapy.Spider):
    name = 'alabama'
    allowed_domains = ['legacy.com']
    start_urls = ['http://www.legacy.com/obituaries/annistonstar/browse?type=paid&page=20']

    def parse(self, response):
    name = response.xpath('//a[@class="NonMobile"]/p[@class="obitName"]/text()').extract()
    link = response.xpath('//div[@class="RightColumn"]//a[@class="ObituaryButton"]/@href').extract()
    obit = response.xpath('//div[@class="NameAndLocation"]/p[@class="obitText"]/text()').extract()
    news = response.xpath('//div[@class="PublishedLine publishedLine"]/span/text()').extract()
    #pull = time.strftime("%m/%d/%Y")
    #state = "AL"

    for item in zip(name, link, obit, news): #removed 'pull, state'

        new_item = AlItem()

        new_item['name'] = item[0]
        new_item['link'] = item[1]
        new_item['obit'] = item[2]
        new_item['news'] = item[3]
        #new_item['pull'] = pull
        #new_item["state"] = state

        yield new_item

Answer 1

我解释原因：

如果在此处粘贴for item in zip(name, link, obit, news): pull＆state，则迭代次数将等于 2 ，因为state = "AL"-字符串变量。 ZIP函数从state得到两个字符，并为循环中的所有参数设置了迭代= 2。 zip从参数中获取最小的麻木进行迭代。与日期01/01/2001一样-10个字符。（迭代次数等于 10 ）

将工作：

`class AlItem(scrapy.Item):
  name = scrapy.Field()
  link = scrapy.Field()
  obit = scrapy.Field()
  news = scrapy.Field()
  pull = scrapy.Field()
  state = scrapy.Field()`

class AlabamaSpider(scrapy.Spider): name = 'alabama' allowed_domains = ['legacy.com'] start_urls = ['http://www.legacy.com/obituaries/annistonstar/browsetype=paid&page=20']

    def parse(self, response):
        name = response.xpath('//a[@class="NonMobile"]/p[@class="obitName"]/text()').extract()
        link = response.xpath('//div[@class="RightColumn"]//a[@class="ObituaryButton"]/@href').extract()
        obit = response.xpath('//div[@class="NameAndLocation"]/p[@class="obitText"]/text()').extract()
        news = response.xpath('//div[@class="PublishedLine publishedLine"]/span/text()').extract()
        pull = time.strftime("%m/%d/%Y")
        state = "AL"

        for item in zip(name, link, obit, news): #removed 'pull, state'

            new_item = AlItem()

            new_item['name'] = item[0]
            new_item['link'] = item[1]
            new_item['obit'] = item[2]
            new_item['news'] = item[3]
            new_item['pull'] = pull
            new_item["state"] = state

            yield new_item

创建静态列项目

1 个答案: