Question

当尝试发送大量请求（＆gt; 500）时，我收到如下连接错误：

ConnectionError :(＆＃39;连接已中止。＆＃39;，gaierror（8，＆＃39;提供节点名称或服务名称，或未知＆＃39;））

代码：

   def compare_and_fix(mac, reqSession):

        url = hosturl + <api as a string> + mac

        try:
                resp = json.loads(reqSession.get(url, headers=headers, cert=(…), timeout=10, verify=False).text)
                response_code = resp['status']
                if response_code == 200:
                        result = resp['data']
                else:
                        print “oo-ooh!”
        except Exception as ex:
                print ex.message


    def worker(reqSession):
            while True:
                    mac = q.get()
                    if validate_mac(mac):
                            compare_and_fix(mac, reqSession)
                    q.task_done()

    num_worker_threads=500
    q = Queue()

    for i in range(num_worker_threads):
            session = requests.session().client()
            t = Thread(target=worker, args=(session,))
            t.daemon = True
            t.start()

    for mac in mac_list:
            q.put(mac)
    print "Waiting for threads to join"                                                                                                                                                                 
    q.join()

如果num_worker_threads低~100-200，这很有效。还有更多，我开始看到上面的错误。如果我超过500，我发现它发生了很多（此时所有调用中约有15％因此错误而失败）。

跟踪：

    exception : Traceback (most recent call last):   
        File “xyz.py", line 62, in abc
                resp = json.loads(reqSession.get(url, headers=headers, cert=(…), timeout=10, verify=False).text)   
        File "/Library/Python/2.7/site-packages/requests/sessions.py", line 476, in get
                return self.request('GET', url, **kwargs)   
        File "/Library/Python/2.7/site-packages/requests/sessions.py", line 464, in request
                resp = self.send(prep, **send_kwargs)   
        File "/Library/Python/2.7/site-packages/requests/sessions.py", line 576, in send
                r = adapter.send(request, **kwargs)   
        File "/Library/Python/2.7/site-packages/requests/adapters.py", line 415, in send 
    ConnectionError: ('Connection aborted.', gaierror(8, 'nodename nor servname provided, or not known'))

对此问题的任何见解表示赞赏。谢谢！

Python - 发送大量请求时出现连接错误

0 个答案: