Question

我现在被困了一段时间，希望有人可以帮助我解决以下问题：

我要做什么：我正尝试从以下网站https://www.coop.nl/boodschappen访问

1）类别级别：

category_url = response.xpath("//div[contains(@class,'block categories')]//a/@href")

2）产品概述级别：

product_url = response.xpath("//article[contains(@class,'gi b0_12 b2_06 b4_08 listItem')]//div/a/@href").extract()

3）产品详细程度：

提取并存储以下项目：

类别级别：category_name，category_url
产品概述级别：product_url
产品详细级别：product_name，product_description， product_pricesI，product_pricesII

当前出现以下错误

line 21, in parse item['category_name']   = category_name 
NameError: name 'item' is not defined

我正在使用Scrapy 1.5.2和Python 3.6.4

Items.py文件

import scrapy

class CoopItem(scrapy.Item):
    category_name = scrapy.Field()
    category_url = scrapy.Field()

    product_name = scrapy.Field()
    product_description = scrapy.Field()
    product_pricesI = scrapy.Field()
    product_pricesII = scrapy.Field()
    product_url = scrapy.Field()

coop.py文件

# -*- coding: utf-8 -*-
import scrapy
from scrapy.http import Request
from ..items import CoopItem

class CoopSpider(scrapy.Spider):
    name = 'coop'
    allowed_domains = ['coop.nl']  
    start_urls = [
        'https://www.coop.nl/boodschappen/'
    ]

    def parse(self, response):
        # categories
        #item = CoopItem()
        category_name = response.xpath("//div[contains(@class,'block categories')]//div[contains(@class,'title')]/span/text()")
        category_url = response.xpath("//div[contains(@class,'block categories')]//a/@href")

        #item['category_name']   = category_name
        #item['category_url']    = category_url

        for href in category_url:
            cat_url = href.extract()
            cat_name = category_name.extract()
            yield Request(url = cat_url,
                          callback = self.parse_products, meta= {'category_name':cat_name, 'category_url':cat_url})

    def parse_products(self, response):
        # Product overview page(s)        
        product_url = response.xpath("//article[contains(@class,'gi b0_12 b2_06 b4_08 listItem')]//div/a/@href").extract()

        #NEXT_PAGE_SELECTOR = '.pagination--lister a.next::attr(href)'
            #next_page = response.css(NEXT_PAGE_SELECTOR).extract_first()
        NEXT_PAGE_SELECTOR = response.xpath("//div[contains(@class,'pagination--lister')]//a[contains(@rel,'next')]/@href")
        for href in NEXT_PAGE_SELECTOR:
            next_page_url = href.extract()
            yield scrapy.Request(url=next_page_url, callback=self.parse_products)

        yield Request(url, callback = self.parse_product_items, meta={'category_name': category_name, 'category_url': category_url, 'product_url': product_url})

    def parse_product_items(self, response):
        # Product detail page

        product_name = response.xpath("//h1[contains(@itemprop,'name')]//text()").extract()
        product_description = response.xpath("//dl[contains(@class,'definitionList')]//text()").extract()
        product_pricesI = response.xpath("//div[contains(@class,'primeDetails gi b0_12 b3_12 b3_push_01 m-0')]//ins/text()").extract()
        product_pricesII = response.xpath("//div[contains(@class,'primeDetails gi b0_12 b3_12 b3_push_01 m-0')]//span/text()").extract_first()

        item = CoopItem()
        item['category_name']       = category_name
        item['category_url']        = category_url
        item['product_name']        = product_name
        item['product_description'] = product_description
        item['product_pricesI']     = product_pricesI
        item['product_pricesII']    = product_pricesII
        item['product_url']         = product_url
        yield item

Answer 1

在parse函数中，您注释了此代码，因此不应出现此错误。
您需要更新其他两个功能：

从meta中提取变量：

def parse_products(self, response):
    # Product overview page(s)     
    category_name = response.meta['category_name']
    category_url = response.meta['category_url']
    ....

def parse_product_items(self, response):
    # Product detail page
    category_name = response.meta['category_name']
    category_url = response.meta['category_url']
    product_url = response.meta['product_url']
    ....

提取3个级别的深层产品详细信息。收到错误NameError：未定义名称“ item”

1 个答案: