Question

我正在使用Scrapy抓取一个新闻网站，并使用sqlalchemy将已删除的项目保存到数据库中。抓取作业会定期运行，我想忽略自上次抓取以来未发生更改的网址。

我正在尝试子类化LinkExtractor并返回一个空列表，以防响应.edl已被更新，而不是更新。

但是当我运行'scrapy crawl spider_name'时，我得到了：

TypeError：MyLinkExtractor（）获得了一个意外的关键字参数 '允许'

代码：

def MyLinkExtractor(LinkExtractor):
    '''This class should redefine the method extract_links to
    filter out all links from pages which were not modified since
    the last crawling'''
    def __init__(self, *args, **kwargs):
        """
        Initializes database connection and sessionmaker.
        """
        engine = db_connect()
        self.Session = sessionmaker(bind=engine)
        super(MyLinkExtractor, self).__init__(*args, **kwargs)

    def extract_links(self, response):
        all_links = super(MyLinkExtractor, self).extract_links(response)

        # Return empty list if current url was recently crawled
        session = self.Session()
        url_in_db = session.query(Page).filter(Page.url==response.url).all()
        if url_in_db and url_in_db[0].last_crawled.replace(tzinfo=pytz.UTC) > item['header_last_modified']:
            return []

        return all_links

...

class MySpider(CrawlSpider):

    def __init__(self, *args, **kwargs):
        """
        Initializes database connection and sessionmaker.
        """
        engine = db_connect()
        self.Session = sessionmaker(bind=engine)
        super(MySpider, self).__init__(*args, **kwargs)

    ...

    # Define list of regex of links that should be followed
    links_regex_to_follow = [
        r'some_url_pattern',
        ]

    rules = (Rule(MyLinkExtractor(allow=links_regex_to_follow),
                  callback='handle_news',
                  follow=True),    
             )

    def handle_news(self, response):

        item = MyItem()
        item['url'] = response.url
        session = self.Session()

        # ... Process the item and extract meaningful info

        # Register when the item was crawled
        item['last_crawled'] = datetime.datetime.utcnow().replace(tzinfo=pytz.UTC)

        # Register when the page was last-modified
        date_string = response.headers.get('Last-Modified', None).decode('utf-8')
        item['header_last_modified'] = get_datetime_from_http_str(date_string)

        yield item

最奇怪的是，如果我在规则定义中替换 LinkExtractor 的 MyLinkExtractor ，它就会运行。

但如果我在规则定义中保留 MyLinkExtractor 并将 MyLinkExtractor 重新定义为：

def MyLinkExtractor(LinkExtractor):
    '''This class should redefine the method extract_links to
    filter out all links from pages which were not modified since
    the last crawling'''
    pass

我得到了同样的错误。

Answer 1

您的MyLinkExtractor不是class，而是功能，因为您已使用def而不是class声明了它。很难发现，因为Python允许在其他函数中声明函数，并且没有任何名称真正被保留。

无论如何，我相信如果没有正确实例化的类，堆栈跟踪会有所不同 - 你会看到错误的最后一个函数的名称（MyLinkExtractor＆＃39; s __init__ ）。

Scrapy子类化LinkExtractor引发TypeError：MyLinkExtractor（）得到一个意外的关键字参数'allow'

1 个答案: