Python多处理(池映射)冻结

时间:2014-10-12 17:58:07

标签: python python-2.7 multiprocessing python-requests

multi()在活动中间某处冻结:

def current_proc(): print mp.current_process().name, 'started'

def multi(fn, func):
   print 'Process started on',time.strftime('%H:%M:%S')

   count = mp.cpu_count()*2
   input = nohead(xlsx2array(fn))
   parts = chunks(input, 10)

   pool = mp.Pool(processes = count, initializer = current_proc, maxtasksperchild = 1)
   for part in parts:
       with stopwatch() as r: pool.map(func, part)

   return r
   pool.close()
   pool.join()

我正在使用多处理功能来获得有效的网址:

def query(i):
    attempts = 2
    while attempts:
        try:
            q = requests.get(i, allow_redirects = True, verify = False, timeout = 2)
            match = q.url                                                                                         
            match = str(match)
            break
        except:
            attempts -= 1
            match = 'pattern not found'
            pass

    return [ i, match ]

请告知我怎样才能避免这种冻结。谢谢,

0 个答案:

没有答案