阻止在进程中调用的多处理pool.map

时间:2017-09-04 19:24:23

标签: python parallel-processing multiprocessing

我想在流程中调用multiprocessing.pool.map

run()函数内初始化时,它可以正常工作。在实例化时初始化时,它不会。

我无法弄清楚这种行为的原因?这个过程会发生什么? 我在python 3.6

from multiprocessing import Pool, Process, Queue

def DummyPrinter(key):
    print(key)

class Consumer(Process):
    def __init__(self, task_queue):
        Process.__init__(self)
        self.task_queue = task_queue
        self.p = Pool(1)

def run(self):
    p = Pool(8)
    while True:
        next_task = self.task_queue.get()
        if next_task is None:
            break

        p.map(DummyPrinter, next_task) #  Works
        #self.p.map(DummyPrinter, next_task)   #  Does not Work
    return

if __name__ == '__main__':
    task_queue = Queue()
    Consumer(task_queue).start()

    task_queue.put(range(5))
    task_queue.put(None)

1 个答案:

答案 0 :(得分:2)

multiprocessing.Pool不能由多个进程共享,因为它依赖于管道和线程来运行。

__init__方法在父进程中执行,而run逻辑属于子进程。

我通常建议不要对Process对象进行子类化,因为它非常直观。

如下所示的逻辑将更好地显示实际的责任分工。

def function(task_queue):
    """This runs in the child process."""
    p = Pool(8)
    while True:
        next_task = self.task_queue.get()
        if next_task is None:
            break

        p.map(DummyPrinter, next_task) #  Works

def main():
    """This runs in the parent process."""
    task_queue = Queue()
    process = Process(target=function, args=[task_queue])
    process.start()