Question

我正在尝试实现一个简单的多处理，其中dict被填充为一组值。函数process基于dict2进行一些简单的计算。该函数如下所示：

def process_control_number(fid, dict1, dict2):

    multiprocessing.current_process().authkey = b"passkey"
    print (fid)
    dict1[fid] = process(dict2[fid])

首先，我尝试了以下实现：

def main():
    manager = Manager()
    dict1 = manager.dict()
    dict2 = getdict2()

    job = [Process(target=process_control_number, args=(fid, dict1, dict2)) for fid in all_files]


    _ = [p.start() for p in job]
    _ = [p.join() for p in job]

但这实际上太慢了（有没有人知道为什么？它每隔2-3秒打印一个ID）。我甚至尝试删除计算并打印fid，但它仍然超级慢，所以我尝试以不同的方式实现它：

def main():
    manager = Manager()
        dict1 = manager.dict()
    dict2 = getdict2()

    pool = multiprocessing.Pool(6) 
    for fid in all_files:
        pool.apply_async(process_control_number, args=(fid, dict1, dict2))
    pool.close()
    pool.join()

但我得到以下错误：

Process SpawnPoolWorker-4:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
    self.run()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\pool.py", line 108, in worker
    task = get()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\queues.py", line 345, in get
    return _ForkingPickler.loads(res)
  File "C:\ProgramData\Anaconda3\lib\site-packages\multiprocess-0.70.5-py3.6.egg\multiprocess\managers.py", line 837, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "C:\ProgramData\Anaconda3\lib\site-packages\multiprocess-0.70.5-py3.6.egg\multiprocess\managers.py", line 691, in __init__
    self._incref()
  File "C:\ProgramData\Anaconda3\lib\site-packages\multiprocess-0.70.5-py3.6.egg\multiprocess\managers.py", line 741, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "C:\ProgramData\Anaconda3\lib\site-packages\multiprocess-0.70.5-py3.6.egg\multiprocess\connection.py", line 496, in Client
    answer_challenge(c, authkey)
  File "C:\ProgramData\Anaconda3\lib\site-packages\multiprocess-0.70.5-py3.6.egg\multiprocess\connection.py", line 742, in answer_challenge
    raise AuthenticationError('digest sent was rejected')
multiprocess.context.AuthenticationError: digest sent was rejected

有人可以帮忙吗？

python 3中奇怪的多处理行为

0 个答案: