Question

问题在于，如果我将产品网址直接添加到＆＃34; start_urls＆＃34;一切正常。但是当抓取期间出现产品页面时（所有已抓取的网页返回＆＃39; 200＆＃39;）它不会刮... 我正在通过蜘蛛：

scrape crawl site_products -t csv -o Site.csv

蜘蛛代码：

#-*- coding: utf-8 -*-
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor
from site.items import SiteItem
import datetime


class SiteProducts(CrawlSpider):
    name = 'site_products'
    allowed_domains = ['www.example.com']
    start_urls = [
        #'http://www.example.com/us/sweater_cod39636734fs.html',
        #'http://www.example.com/us/sweater_cod39693703uh.html',
        #'http://www.example.com/us/pantaloni-5-tasche_cod36883777uu.html',
        #'http://www.example.com/fr/robe_cod34663996xk.html',
        #'http://www.example.com/fr/trousers_cod36898044mj.html',
        'http://www.example.com/us/women/onlinestore/suits-and-jackets',
    ]

    rules = (
        # Extract links matching 'item.php' and parse them with the spider's method parse_item
        Rule(LinkExtractor(allow=('http://www.example.com/us/', 'http://www.example.com/fr/', )), follow=True),
        Rule(LinkExtractor(allow=('.*_cod.*\.html', )), callback='parse_item'),
    )

    def parse_item(self, response):
        self.logger.info('Hi, this is an item page! %s', response.url)
        item = SiteItem()
        item['name'] = response.xpath('//h2[@class="productName"]/text()').extract()
        item['price'] = response.xpath('//span[@class="priceValue"]/text()')[0].extract()
        if response.xpath('//span[@class="currency"]/text()')[0].extract() == '$':
            item['currency'] = 'USD'
        else:
            item['currency'] = response.xpath('//span[@class="currency"]/text()')[0].extract()
        item['category'] = response.xpath('//li[@class="selected leaf"]/a/text()').extract()
        item['sku'] = response.xpath('//span[@class="MFC"]/text()').extract()
        if response.xpath('//div[@class="soldOutButton"]/text()').extract() == True or response.xpath('//span[@class="outStock"]/text()').extract() == True:
            item['avaliability'] = 'No'
        else:
            item['avaliability'] = 'Yes'
        item['time'] = datetime.datetime.now().strftime("%Y.%m.%d %H:%M")
        item['color'] = response.xpath('//*[contains(@id, "color_")]/a/text()').extract()
        item['size'] = response.xpath('//*[contains(@id, "sizew_")]/a/text()').extract()
        if '/us/' in response.url:
            item['region'] = 'US'
        elif '/fr/' in response.url:
            item['region'] = 'FR'
        item['description'] = response.xpath('//div[@class="descriptionContent"]/text()')[0].extract()
        return item

我错过了什么？

Answer 1

我已经过测试，似乎该网站会阻止所有非标准用户代理（通过返回403）。因此，请尝试将user_agent类参数设置为常见的类似：

class SiteProducts(CrawlSpider):
    name = 'site_products'
    user_agent = 'Mozilla/5.0 (X11; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0'

或只在项目settings.py中设置：

USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0'

您可以在网络上找到更多用户代理字符串，例如official mozzila docummentation

编辑：
经过进一步检查，我发现你的LinkExtractor逻辑有问题。 Linkextractors以定义的规则顺序提取并且提取器覆盖过度，这意味着第一个带有follow的linkextractor也会提取产品页面，这意味着您拥有的产品链接提取器将抓取之前已经爬行的页面并获得dupe过滤。

您需要重新设计第一个linkextractor以避免产品页面。您只需将allow参数从linkextractor复制到第一个linkextractor的deny参数即可。

Scrapy爬行但不刮

1 个答案: