如果生成的线程多于池大小,为什么gevent池会中断?

时间:2014-03-29 18:14:08

标签: python multithreading csv gevent greenlets

我希望下面的代码能够使用10个并发线程,并在池释放后完成其余的操作。相反,额外的调用导致崩溃的线程,就好像被调用的函数试图继续进行,即使urllib请求没有成功。但为什么?我认为在池打开greenlet之前不会调用整个函数。或者即使函数被提前调用,是什么阻止它完成?

FWIW:workrow函数(未显示)获取列表(CSV的一行),在Web上发出API请求,解析JSON,并在另一个CSV中写入一行。

import gevent
from gevent import socket
from gevent import monkey, pool
# patches stdlib (including socket and ssl modules) to cooperate with other greenlets
monkey.patch_all()
p = pool.Pool(10)

with open(inputfile, 'rb') as csvfile:
    entreader = unicodecsv.reader(csvfile, delimiter=',', quotechar='"')
    head=[entreader.next() for x in xrange(20)]
    for row in head:
        p.spawn(workrow, row)
p.join()

错误:

Traceback (most recent call last):
  File "/Users/laszlosandor/anaconda/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/Users/laszlosandor/Downloads/GoogleGeocoding/google_geocoding_LS.py", line 37, in workrow
    result = data_json["results"][0]
IndexError: list index out of range
<Greenlet at 0x101245c30: workrow([u'p_366937', u'/entity/p/366937.xml', u'H\xe9derv)> failed with IndexError

0 个答案:

没有答案