使用DNSCACHE_ENABLED进行scrapy = False无法正常工作

时间:2017-07-03 03:16:10

标签: scrapy scrapy-shell

当我使用DNSCACHE_ENABLED运行scrapy shell时False得到KeyError:'字典为空' twisted.internet.error.DNSLookupError:DNS查找失败:主机名查找没有结果:www.mydomain.com。

2017-07-03 03:09:12 [twisted] CRITICAL: while looking up www.mydomain.com with <scrapy.resolver.CachingThreadedResolver object at 0x3fd0050>
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/twisted/internet/defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib64/python2.7/site-packages/scrapy/resolver.py", line 29, in _cache_result
    dnscache[name] = result
  File "/usr/lib64/python2.7/site-packages/scrapy/utils/datatypes.py", line 305, in __setitem__
    self.popitem(last=False)
  File "/usr/lib64/python2.7/collections.py", line 159, in popitem
    raise KeyError('dictionary is empty')
KeyError: 'dictionary is empty'
2017-07-03 03:09:12 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET //www.mydomain.com/> (failed 3 times): DNS lookup failed: no results for hostname lookup: www.mydomain.com.
Traceback (most recent call last):
  File "/usr/bin/scrapy", line 11, in <module>
    sys.exit(execute())
  File "/usr/lib64/python2.7/site-packages/scrapy/cmdline.py", line 149, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/usr/lib64/python2.7/site-packages/scrapy/cmdline.py", line 89, in _run_print_help
    func(*a, **kw)
  File "/usr/lib64/python2.7/site-packages/scrapy/cmdline.py", line 156, in _run_command
    cmd.run(args, opts)
  File "/usr/lib64/python2.7/site-packages/scrapy/commands/shell.py", line 73, in run
    shell.start(url=url, redirect=not opts.no_redirect)
  File "/usr/lib64/python2.7/site-packages/scrapy/shell.py", line 48, in start
    self.fetch(url, spider, redirect=redirect)
  File "/usr/lib64/python2.7/site-packages/scrapy/shell.py", line 115, in fetch
    reactor, self._schedule, request, spider)
  File "/usr/lib64/python2.7/site-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "<string>", line 2, in raiseException
twisted.internet.error.DNSLookupError: DNS lookup failed: no results for hostname lookup: www.mydomain.com.

欢迎任何想法

0 个答案:

没有答案