Question

我在专用服务器上使用scrapy并且我多次出现此错误：

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
  File "dirbot/spiders/expired.py", line 54, in errback_httpbin
  File "dirbot/spiders/expired.py", line 65, in checkDomain
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
  File "/usr/lib/python2.7/urllib2.py", line 431, in open
  File "/usr/lib/python2.7/urllib2.py", line 449, in _open
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
  File "/usr/lib/python2.7/urllib2.py", line 1227, in http_open
  File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
URLError: <urlopen error [Errno 24] Too many open files>

我已经添加了此配置，但我仍然遇到此错误。

at /etc/sysctl.conf

添加：
net.core.somaxconn=131072
fs.file-max=131072
然后：
sudo sysctl -p
at /usr/include/linux/limits.h

变化：
NR_OPEN = 65536
at /etc/security/limits.conf

添加：
*                soft    nofile          65535
*                hard    nofile          65535

有人可以帮我解决这个问题吗？

更新，我的配置Scrapy：

custom_settings = {
    'RETRY_ENABLED': False,
    'DEPTH_LIMIT' : 0,
    'DEPTH_PRIORITY' : 1,
    'LOG_ENABLED' : False,
    'CONCURRENT_REQUESTS_PER_DOMAIN' : 64,
    'CONCURRENT_REQUESTS' : 128,
    'REACTOR_THREADPOOL_MAXSIZE' : 30,
    'COOKIES_ENABLED' : False,
    'DOWNLOAD_TIMEOUT' : 10,
    'DOWNLOAD_WARNSIZE' : 66554432
}

URLError：<urlopen error =“”[errno =“”24] =“”too =“”many =“”open =“”files =“”>

0 个答案: