Question

我正在尝试使用urilib2查询api时加快我的程序。为此，我尝试用多线程编写。

然而，当我运行代码时，它会出现以下错误，并且我没有通过urlib2调用得到任何响应。

Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
  File "/usr/lib64/python2.6/threading.py", line 484, in run
  File "./myscript", line 333, in query_api
  File "./myscript", line 350, in issue_request
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
  File "/usr/lib64/python2.6/urllib2.py", line 1164, in do_open
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Exception in thread Thread-2 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib64/python2.6/threading.py", line 532, in __bootstrap_inner
  File "/usr/lib64/python2.6/threading.py", line 484, in run
  File "./myscript", line 333, in query_api
  File "./myscript", line 350, in issue_request
  File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen
  File "/usr/lib64/python2.6/urllib2.py", line 391, in open
  File "/usr/lib64/python2.6/urllib2.py", line 409, in _open
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
  File "/usr/lib64/python2.6/urllib2.py", line 1198, in https_open
  File "/usr/lib64/python2.6/urllib2.py", line 1164, in do_open
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

我的代码是：

class ApiMultiThreadHelper:

    def __init__(self,api_calls):
        self.q = Queue.Queue()
        self.api_datastore = {}
        self.api_calls = api_calls
        self.userpass = '#####'

    def query_api(self,q,api_query):
        self.q.put(self.issue_request(api_query))

    def issue_request(self,api_query):

        self.api_datastore.update({api_query:{}})

        for lookup in ["call1","call2"]:
            query = api_query+lookup

            request = urllib2.Request(query)
            request.add_header("Authorization", "Basic %s" % self.userpass)
            f = urllib2.urlopen(request)
            response = f.read()
            f.close()

            self.api_datastore[api_query].update({lookup:response})

        return True

    def go(self):
        for i in self.api_calls:
            t = threading.Thread(target=self.query_api, args = (self.q,i))
            t.daemon = True
            t.start()

        return self.api_datastore

谢谢，

Answer 1

urllib2应该是线程安全的。这是猜测，但是......

线程Thread-2中的异常（很可能在解释器关闭期间引发）

这表明你不是在等待你的线程完成这项工作。有问题的一行就是这个：

t.daemon = True

结果是当主线程退出时，所有内容都被终止（主线程将只等待非守护进程）。尝试删除该行。你也应该正确加入：

threads = []
for i in self.api_calls:
    t = threading.Thread(target=self.query_api, args = (self.q,i))
    t.start()
    threads.append(t)

for t in threads:
    t.join()  # <--- wait for all threads to finish their jobs

使用urllib2进行多线程处理

1 个答案: