“批处理”多处理队列作业

Question

有没有一种方法可以将多个项目从队列启动到工作池？

我有一个生产者在队列中填充要处理的项目，一个消费者在后台运行以等待任务。

我想让消费者同时从事多个工作：

import multiprocessing as mp
import time

def producer(queue):
    print "produce"
    for i in range(0,100,5):
        for j in range(i,i+5,1):
            queue.put(i)
            print "Producer put: %s" % i
        time.sleep(1)
    queue.put(None)


def consumer2(queue):
    my_pool = mp.Pool(5)
    init_size = queue.qsize()
    for i in range(0, init_size, 5):
        items = [queue.get() for k in range(0,4)]
        my_pool.map(test_worker, items)
    my_pool.close()
    my_pool.join()

def test_worker(inp):
    print "Received %s as input" % inp
    time.sleep(0.4)
    return

if __name__ == '__main__':
    pool = mp.Pool(3)
    data_q = mp.JoinableQueue()
    p = pool.Process(target=consumer2, args=(data_q, ))
    c = pool.Process(target=producer, args=(data_q, ))
    c.start()
    p.start()
    p.join()
    c.join()

这是我正在观察的输出：

produce
Producer put: 0
Producer put: 0
Producer put: 0
Producer put: 0
Producer put: 0
Received 0 as input
Received 0 as input
Received 0 as input
Received 0 as input
Producer put: 5
Producer put: 5
Producer put: 5
Producer put: 5
Producer put: 5
Producer put: 10
Producer put: 10
Producer put: 10
Producer put: 10
Producer put: 10
Producer put: 15
Producer put: 15
Producer put: 15
Producer put: 15
Producer put: 15
...

使用者似乎在处理了前5个任务后就退出了。 我希望消费者在完成一批任务之后而不是退出后检查队列。