Question

这个使用multiprocessing的简单Python3程序似乎没有按预期工作。

所有输入进程共享一个输入队列，从中消耗数据。它们都共享一个输出队列，一旦完成它们就会写入结果。我发现此程序在进程join()中挂起。那是为什么？

#!/usr/bin/env python3

import multiprocessing

def worker_func(in_q, out_q):
    print("A worker has started")    
    w_results = {}
    while not in_q.empty():
        v = in_q.get()
        w_results[v] = v
    out_q.put(w_results)
    print("A worker has finished")

def main():

    # Input queue to share among processes
    fpaths = [str(i) for i in range(10000)]
    in_q = multiprocessing.Queue()
    for fpath in fpaths:
        in_q.put(fpath)

    # Create processes and start them
    N_PROC = 2
    out_q = multiprocessing.Queue()
    workers = []
    for _ in range(N_PROC):
        w = multiprocessing.Process(target=worker_func, args=(in_q, out_q,))
        w.start()
        workers.append(w)
    print("Done adding workers")

    # Wait for processes to finish
    for w in workers:
        w.join()
    print("Done join of workers")

    # Collate worker results
    out_results = {}
    while not out_q.empty():
        out_results.update(out_q.get())

if __name__ == "__main__":
    main()

我在N_PROC = 2：

时从此计划中获得此结果

$ python3 test.py
Done adding workers
A worker has started
A worker has started
A worker has finished
<---- I do not get "A worker has finished" from second worker
<---- I do not get "Done join of workers"

即使只有一个子进程N_PROC = 1，它也不起作用：

$ python3 test.py
Done adding workers
A worker has started
A worker has finished
<---- I do not get "Done join of workers"

如果我尝试使用较小的输入队列来说1000项，那么一切正常。

我知道一些旧的StackOverflow问题，说Queue有一个限制。为什么Python3文档中没有记录这个？

我可以使用什么替代解决方案？我想使用多处理（而不是线程）来分割N个进程之间的输入。一旦他们的共享输入队列为空，我希望每个进程收集其结果（可以是像dict这样的大/复杂数据结构）并将其返回到父进程。怎么做？

Answer 1

这是由您的设计引起的经典错误。当工作人员终止时，他们会因为无法将所有数据放入out_q而失速，从而使您的程序陷入僵局。这与您队列中的管道缓冲区的大小有关。

当您使用multiprocessing.Queue时，您应该在尝试加入馈送器进程之前将其清空，以确保Process不会停止等待将所有对象放入{ {1}}。因此，在加入流程之前进行Queue调用可以解决您的问题：您可以使用标记模式来检测计算的结束。

out_q.get

另请注意，您的代码中存在竞争条件。在您检查#!/usr/bin/env python3 import multiprocessing from multiprocessing.queues import Empty def worker_func(in_q, out_q): print("A worker has started") w_results = {} while not in_q.empty(): try: v = in_q.get(timeout=1) w_results[v] = v except Empty: pass out_q.put(w_results) out_q.put(None) print("A worker has finished") def main(): # Input queue to share among processes fpaths = [str(i) for i in range(10000)] in_q = multiprocessing.Queue() for fpath in fpaths: in_q.put(fpath) # Create processes and start them N_PROC = 2 out_q = multiprocessing.Queue() workers = [] for _ in range(N_PROC): w = multiprocessing.Process(target=worker_func, args=(in_q, out_q,)) w.start() workers.append(w) print("Done adding workers") # Collate worker results out_results = {} n_proc_end = 0 while not n_proc_end == N_PROC: res = out_q.get() if res is None: n_proc_end += 1 else: out_results.update(res) # Wait for processes to finish for w in workers: w.join() print("Done join of workers") if __name__ == "__main__": main()和in_q之间，可以清空队列not in_q.empty()。您应该使用非阻塞get来确保您没有死锁，等待空队列。

最后，您正在尝试实现类似get的内容，以更强大的方式处理此类通信。您还可以查看concurrent.futures API，它更加强大，在某种意义上说，设计得更好。

为什么没有输入和输出队列的Python进程一旦完成就不会加入？

1 个答案: