Question

我想将数据发送到多个进程。每个进程都对该数据执行其他操作并等待下一个数据。我有这样的事情：

from multiprocessing import Process, Manager

def do_work1(in_queue):
    while True:
        item = in_queue.get()
        # exit signal
        if item == None:
            return
        print "worker 1 : {}".format(item)

def do_work2(in_queue):
    while True:
        item = in_queue.get()
        # exit signal
        if item == None:
            return
        print "worker 2: {}".format(item)



if __name__ == "__main__":
    num_workers = 2

    manager = Manager()

    work = manager.Queue(num_workers)

    # start for workers
    pool = []
    p = Process(target=do_work1, args=(work,))
    p.start()
    pool.append(p)
    p2 = Process(target=do_work2, args=(work,))
    p2.start()
    pool.append(p2)



    work.put("1")
    work.put("2")

    for p in pool:
        p.join()

但是在运行此代码之后我得到了：

worker 1 : 1

worker 1 : 2

或者：

worker 2 : 1

worker 1 : 2

我期待这样的事情：

worker 1 : 1

worker 2 : 1

worker 1 : 2

worker 2 : 2

如果我想获得上述结果，我应该改变什么？

Answer 1

您可以使用多处理管道将数据发送到每个进程：

https://docs.python.org/2/library/multiprocessing.html#multiprocessing.Pipe

假设数据能够被腌制：

https://docs.python.org/2/library/pickle.html#what-can-be-pickled-and-unpickled

您可以让进程等待管道末端的数据，并在发送数据时循环遍历所有管道的列表。

Answer 2

您可以使用multiprocessing.Pool

来消除手动队列处理

import multiprocessing


def do_work1(item):
    print "worker 1: {}".format(item)


def do_work2(item):
    print "worker 2: {}".format(item)


if __name__ == '__main__':
    pool = multiprocessing.Pool(2)

    pool.apply_async(do_work1, (1,))
    pool.apply_async(do_work2, (1,))
    pool.apply_async(do_work1, (2,))
    pool.apply_async(do_work2, (2,))

    pool.close()
    pool.join()

在Python 2中将数据发送到多个进程

2 个答案: