我正在尝试执行大约100k的GET请求并解析每个请求的响应主体。我认为问候是一个很好的方法,但我得到的错误与'太多的打开文件'有关。这是代码:
import grequests
with open("./100k-sites.csv", "r") as f:
urls = ["http://" + line.rstrip() for line in f]
rs = (grequests.get(u, timeout=1) for u in urls)
responses = grequests.map(rs)
for r in responses:
try:
# do something with the response body
except:
pass
谁有经验呢?我得到的错误是:
requests.packages.urllib3.connection.HTTPConnection对象at 0x7f817ab36898>:无法建立新连接[Errno 24] Too 许多打开的文件
答案 0 :(得分:1)
也许这只是一种解决方法(正如上面提到的discussion中的某些人所说的那样),但恕我直言,这里有值得写的,可以用两行来修复它:
import resource
resource.setrlimit(resource.RLIMIT_NOFILE, (110000, 110000))