上下文
我需要在multiprocessing.ThreadPool中运行multiprocessing.Process。 起初看起来很奇怪，但这是我发现处理segfault的唯一方法，因为我使用的是c ++共享库。 如果追加了段错误，则该进程被终止，我可以检查process.exitcode并处理它。
问题
过了一会儿，当我试图加入这个过程时会出现死锁。
以下是我的代码的简单版本：
import sys, time, multiprocessing
from multiprocessing.pool import ThreadPool
def main():
# Launch 8 workers
pool = ThreadPool(8)
it = pool.imap(run, range(500))
while True:
try:
it.next()
except StopIteration:
break
def run(value):
# Each worker launch it own Process
process = multiprocessing.Process(target=run_and_might_segfault, args=(value,))
process.start()
while process.is_alive():
sys.stdout.write('.')
sys.stdout.flush()
time.sleep(0.1)
# Will never join after a while, because of a mystery deadlock
process.join()
# Deals with process.exitcode to log errors
def run_and_might_segfault(value):
# Load a shared library and do stuff (could throw c++ exception, segfault ...)
print(value)
if __name__ == '__main__':
main()
这是一个可能的输出：
➜ ~ python m.py
..0
1
........8
.9
.......10
......11
........12
13
........14
........16
........................................................................................
正如您所看到的，process.is_alive()在几次迭代后总是如此，该过程永远不会加入。
如果我按CTRL-C脚本来获取此stacktrace：
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/multiprocessing/pool.py", line 680, in next
item = self._items.popleft()
IndexError: pop from an empty deque
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "m.py", line 30, in <module>
main()
File "m.py", line 9, in main
it.next()
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5 /lib/python3.5/multiprocessing/pool.py", line 684, in next
self._cond.wait(timeout)
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5 /lib/python3.5/threading.py", line 293, in wait
waiter.acquire()
KeyboardInterrupt
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5 /lib/python3.5/multiprocessing/popen_fork.py", line 29, in poll
pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
PS 在macos上使用python 3.5.2。
感谢各种帮助。
修改
我尝试使用python 2.7，它运行良好。可能只是一个python 3.5问题？
答案 0 :(得分：6)
问题也在CPython的最新版本中再现 - Python 3.7.0a0 (default:4e2cce65e522, Oct 13 2016, 21:55:44)。
如果您使用gdb attach进入其中一个卡住的进程，您会发现它正在sys.stdout.flush()调用中尝试获取锁定：
(gdb) py-list
263 import traceback
264 sys.stderr.write('Process %s:\n' % self.name)
265 traceback.print_exc()
266 finally:
267 util.info('process exiting with exitcode %d' % exitcode)
>268 sys.stdout.flush()
269 sys.stderr.flush()
270
271 return exitcode
Python级别的回溯看起来像这样：
(gdb) py-bt
Traceback (most recent call first):
File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/process.py", line 268, in _bootstrap
sys.stdout.flush()
File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/popen_fork.py", line 74, in _launch
code = process_obj._bootstrap()
File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/context.py", line 277, in _Popen
return Popen(process_obj)
File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/context.py", line 223, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "deadlock.py", line 17, in run
process.start()
File "/home/rpodolyaka/src/cpython/Lib/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "/home/rpodolyaka/src/cpython/Lib/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "/home/rpodolyaka/src/cpython/Lib/threading.py", line 916, in _bootstrap_inner
self.run()
File "/home/rpodolyaka/src/cpython/Lib/threading.py", line 884, in _bootstrap
self._bootstrap_inner()
在翻译级别，它看起来像：
(gdb) frame 6
(gdb) list
287 return 0;
288 }
289 relax_locking = (_Py_Finalizing != NULL);
290 Py_BEGIN_ALLOW_THREADS
291 if (!relax_locking)
292 st = PyThread_acquire_lock(self->lock, 1);
293 else {
294 /* When finalizing, we don't want a deadlock to happen with daemon
295 * threads abruptly shut down while they owned the lock.
296 * Therefore, only wait for a grace period (1 s.). ... */
(gdb) p /x self->lock
$1 = 0xd25ce0
(gdb) p /x self->owner
$2 = 0x7f9bb2128700
注意，从这个特定子进程的角度来看，锁仍然由父进程中的一个线程（LWP 1105）拥有：
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f9bb5559440 (LWP 1102) "python" 0x00007f9bb5157577 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0,
futex_word=0xe4d340) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
2 Thread 0x7f9bb312a700 (LWP 1103) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
3 Thread 0x7f9bb2929700 (LWP 1104) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
4 Thread 0x7f9bb2128700 (LWP 1105) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
5 Thread 0x7f9bb1927700 (LWP 1106) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
6 Thread 0x7f9bb1126700 (LWP 1107) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
7 Thread 0x7f9bb0925700 (LWP 1108) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
8 Thread 0x7f9b9bfff700 (LWP 1109) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
9 Thread 0x7f9b9b7fe700 (LWP 1110) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
10 Thread 0x7f9b9affd700 (LWP 1111) "python" 0x00007f9bb4780253 in select () at ../sysdeps/unix/syscall-template.S:84
11 Thread 0x7f9b9a7fc700 (LWP 1112) "python" 0x00007f9bb5157577 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0,
futex_word=0x7f9b80001ed0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
12 Thread 0x7f9b99ffb700 (LWP 1113) "python" 0x00007f9bb5157577 in futex_abstimed_wait_cancelable (private=0, abstime=0x0, expected=0,
futex_word=0x7f9b84001bb0) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
所以它确实是一个死锁，它发生的原因是你在多个sys.stdout执行写入和刷新
线程同时在原始进程中同时创建子进程 - 根据fork(2)系统调用的性质
子项继承父内存，包括获取的锁：在获取锁时必须执行fork()次调用，即使父进程最终释放它，孩子也不会看到，因为现在每个有自己的内存空间，在写入时被复制。
因此，混音时需要非常小心
具有多处理功能的多线程，并确保在fork()之前正确释放所有锁定（如果必须的话）
用于儿童过程。
它与http://bugs.python.org/issue6721
中描述的内容非常相似请注意，如果您从代码段中移除了与sys.stdout的互动，则会正常运行。