Question

我正在使用scrapy抓取google play项目个人资料，但不执行回调功能。我在代码中找不到问题（没有错误）。你能告诉我任何解决方案吗？

# -*- coding: utf-8 -*-
import scrapy

import time

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from playcrawl.items import PlaycrawlItem
from scrapy.http import Request

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.linkextractor import LinkExtractor

class GoogleplaySpider(CrawlSpider):
    name = 'googleplay'
    allowed_domains = ['play.google.com']
    start_urls = ['https://play.google.com/store/apps/category/GAME']

    rules = (
        Rule(LinkExtractor(allow=('/store/apps'))),
        Rule(LinkExtractor(allow=('/store/apps/details\?')),callback="parse_item")
        )

    def parse_item(self, response):
        hxs = HtmlXPathSelector(response)

        item = PlaycrawlItem()
        item["pub"] = hxs.select('//a[@class = "document-subtitle primary"]/span[1]').select("text()").extract()
        item["email"] = hxs.select('//a[contains(@class, "dev-link") and starts-with(@href, "mailto")]').select("@href").extract()[0][7:]

        f = open("D:\\_scrapy\\playcrawl\\data_emails.txt", "a")
        f.write(item["email"] + "\n")
        f.close()

        print("\n\n\n\n" + item["email"] + "\n\n\n\n")
        time.sleep(0)

        return item #yield item

Answer 1

我测试了你的代码，原因很简单。

蜘蛛刚刚与第二个let result = _.reject(arrayOfObjects, (obj) => _.isEqual(obj, objToCompare))匹配。

试试这个：

Rule

然后它可以工作，所以你的代码没有错误，但关于你的逻辑。

Python / Scrapy：回调函数从不调用

1 个答案: