我将multiprocessing.Pool与tqdm结合使用。我发现有时这种组合喜欢卡住。我该怎么办才能解决这个问题？
例如，我正在运行以下内容：
with mp.Pool(processes=mp.cpu_count()) as p:
list(tqdm(p.imap(create_bins, args),
total=len(args),
desc='Segmentation...'))
进度条一直停留在1％，但是我可以在后台看到create_bins处理了10000个请求的文件/作业中的9961。
我还可以看到Python进程的数量从8或9减少到1，而CPU和内存的消耗却很低。
除了Ctrl + C以外，我无能为力，我得到的唯一诊断信息是：
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/queues.py", line 351, in get
with self._rlock:
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/queues.py", line 351, in get
with self._rlock:
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/queues.py", line 351, in get
with self._rlock:
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/pool.py", line 110, in worker
task = get()
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/queues.py", line 351, in get
with self._rlock:
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/synchronize.py", line 95, in __enter__
return self._semlock.__enter__()
KeyboardInterrupt
Traceback (most recent call last):
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/pool.py", line 733, in next
item = self._items.popleft()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "run_clustering.py", line 180, in <module>
# run_segmentation(imgs_fpaths, args.bins_dir)
File "run_clustering.py", line 146, in run_segmentation
desc='Segmentation...'))
File "/home/my_user/.conda/envs/my_env/lib/python3.7/site-packages/tqdm/std.py", line 1102, in __iter__
for obj in iterable:
File "/home/my_user/.conda/envs/my_env/lib/python3.7/multiprocessing/pool.py", line 737, in next
self._cond.wait(timeout)
File "/home/my_user/.conda/envs/my_env/lib/python3.7/threading.py", line 296, in wait
waiter.acquire()
KeyboardInterrupt
Segmentation...: 1%|▍ | 141/10000 [1:12:58<85:02:17, 31.05s/it]
我在Ubuntu 18.04.3 LTS上使用tqdm 4.40.2，Python 3.7.3。