我的具体用例是： 我有一个刮刮一个网站的刮刀，一旦一个项目产生 - 我有一个绑定信号，在Redis中设置了一个到期时间的密钥。下次运行scraper时，它应该忽略Redis中存在密钥的所有URL。
第一部分我工作得很好;第二部分 - 我创建了一个DownloaderMiddleware，它有一个process_request函数，用于查看传入的请求对象，并检查它的URL是否存在于Redis中。如果是，则raise为IgnoreRequest例外。
我想知道的是： 有没有办法静默出列请求而不是引发异常？ 它不是一个美学要求的硬性要求;我不想在我的错误日志中看到其中的一些 - 我只想看到真正的错误。
我在Scrapy源代码中看到他们在主调度程序（scrapy / core / scheduler.py）中使用看起来像重复过滤的kludge：
def enqueue_request(self, request):
if not request.dont_filter and self.df.request_seen(request):
self.df.log(request, self.spider)
return False
答案 0 :(得分：3)
来自评论的OP的中间件代码
def __init__(self, crawler):
self.client = Redis()
self.crawler = crawler
self.crawler.signals.connect(self.process_request, signals.request_scheduled)
def process_request(self, request, spider):
if not self.client.is_deferred(request.url): # URL is not deferred, proceed as normal
return None
raise IgnoreRequest('URL is deferred')
问题在于您在signals.request_scheduled上添加的信号处理程序。如果它引发异常，it will appear in the logs
我认为在这里注册process_request作为信号处理程序是不正确的（或无意义的）。
我可以使用这个类似（非正确）的测试中间件重现您的控制台错误，该中间件会忽略它看到的所有其他请求：
from scrapy import log, signals
from scrapy.exceptions import IgnoreRequest
class TestMiddleware(object):
def __init__(self, crawler):
self.counter = 0
@classmethod
def from_crawler(cls, crawler):
o = cls(crawler)
crawler.signals.connect(o.open_spider, signals.spider_opened)
# this raises an exception always and will trigger errors in the console
crawler.signals.connect(o.process, signals.request_scheduled)
return o
def open_spider(self, spider):
spider.logger.info('TestMiddleware.open_spider()')
def process_request(self, request, spider):
spider.logger.info('TestMiddleware.process_request()')
self.counter += 1
if (self.counter % 2) == 0:
raise IgnoreRequest("ignoring request %d" % self.counter)
def process(self, *args, **kwargs):
raise Exception
了解使用此中间件运行蜘蛛时控制台所说的内容：
2016-04-06 00:16:58 [scrapy] ERROR: Error caught on signal handler: <bound method ?.process of <mwtest.middlewares.TestMiddleware object at 0x7f83d4a73f50>>
Traceback (most recent call last):
File "/home/paul/.virtualenvs/scrapy11rc3.py27/local/lib/python2.7/site-packages/scrapy/utils/signal.py", line 30, in send_catch_log
*arguments, **named)
File "/home/paul/.virtualenvs/scrapy11rc3.py27/local/lib/python2.7/site-packages/pydispatch/robustapply.py", line 55, in robustApply
return receiver(*arguments, **named)
File "/home/paul/tmp/mwtest/mwtest/middlewares.py", line 26, in process
raise Exception
Exception
与此比较：
$ cat middlewares.py
from scrapy import log, signals
from scrapy.exceptions import IgnoreRequest
class TestMiddleware(object):
def __init__(self, crawler):
self.counter = 0
@classmethod
def from_crawler(cls, crawler):
o = cls(crawler)
crawler.signals.connect(o.open_spider, signals.spider_opened)
return o
def open_spider(self, spider):
spider.logger.info('TestMiddleware.open_spider()')
def process_request(self, request, spider):
spider.logger.info('TestMiddleware.process_request()')
self.counter += 1
if (self.counter % 2) == 0:
raise IgnoreRequest("ignoring request %d" % self.counter)
IgnoreRequest未打印在日志中，但您最终会在统计信息中计算异常：
$ scrapy crawl httpbin
2016-04-06 00:27:24 [scrapy] INFO: Scrapy 1.1.0rc3 started (bot: mwtest)
(...)
2016-04-06 00:27:24 [scrapy] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'mwtest.middlewares.TestMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.chunked.ChunkedTransferMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
(...)
2016-04-06 00:27:24 [scrapy] INFO: Spider opened
2016-04-06 00:27:24 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-04-06 00:27:24 [httpbin] INFO: TestMiddleware.open_spider()
2016-04-06 00:27:24 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-04-06 00:27:24 [httpbin] INFO: TestMiddleware.process_request()
2016-04-06 00:27:24 [httpbin] INFO: TestMiddleware.process_request()
2016-04-06 00:27:24 [httpbin] INFO: TestMiddleware.process_request()
2016-04-06 00:27:24 [httpbin] INFO: TestMiddleware.process_request()
2016-04-06 00:27:24 [httpbin] INFO: TestMiddleware.process_request()
2016-04-06 00:27:24 [scrapy] DEBUG: Crawled (200) <GET http://www.httpbin.org/user-agent> (referer: None)
2016-04-06 00:27:25 [scrapy] DEBUG: Crawled (200) <GET http://www.httpbin.org/> (referer: None)
2016-04-06 00:27:25 [scrapy] DEBUG: Crawled (200) <GET http://www.httpbin.org/headers> (referer: None)
2016-04-06 00:27:25 [scrapy] INFO: Closing spider (finished)
2016-04-06 00:27:25 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 2,
'downloader/exception_type_count/scrapy.exceptions.IgnoreRequest': 2,
'downloader/request_bytes': 665,
'downloader/request_count': 3,
'downloader/request_method_count/GET': 3,
'downloader/response_bytes': 13006,
'downloader/response_count': 3,
'downloader/response_status_count/200': 3,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2016, 4, 5, 22, 27, 25, 596652),
'log_count/DEBUG': 4,
'log_count/INFO': 13,
'log_count/WARNING': 1,
'response_received_count': 3,
'scheduler/dequeued': 5,
'scheduler/dequeued/memory': 5,
'scheduler/enqueued': 5,
'scheduler/enqueued/memory': 5,
'start_time': datetime.datetime(2016, 4, 5, 22, 27, 24, 661345)}
2016-04-06 00:27:25 [scrapy] INFO: Spider closed (finished)
答案 1 :(得分：0)
Scrapy使用Python模块logging来记录事物。你想要的只是一个美学的东西，你可以写一个logging filter来过滤掉你不想看的东西。