Question

我需要将大型列表中的每个对象传递给一个函数。函数完成后，我不再需要传递给函数的对象，并希望删除对象以节省内存。如果我正在使用单个流程，我会执行以下操作：

result = []
while len(mylist) > 0:
    result.append(myfunc(mylist.pop())

当我遍历mylist时，我会弹出列表中的每个对象，以便在将对象传递给我的函数后，该对象不再存储在mylist中。如何使用multiprocessing并行地实现同样的效果？

Answer 1

一个简单的消费者示例（credits go here）：

import multiprocessing
import time
import random

class Consumer(multiprocessing.Process):

    def __init__(self, task_queue, result_queue):
        multiprocessing.Process.__init__(self)
        self.task_queue = task_queue
        self.result_queue = result_queue

    def run(self):
        while True:
            task = self.task_queue.get()
            if task is None:
                # Poison pill means shutdown
                self.task_queue.task_done()
                break
            answer = task.process()
            self.task_queue.task_done()
            self.result_queue.put(answer)
        return


class Task(object):

    def process(self):
        time.sleep(0.1) # pretend to take some time to do the work
        return random.randint(0, 100)


if __name__ == '__main__':
    # Establish communication queues
    tasks = multiprocessing.JoinableQueue()
    results = multiprocessing.Queue()

    # Start consumers
    num_consumers = multiprocessing.cpu_count() * 2
    consumers = [Consumer(tasks, results) for i in xrange(num_consumers)]
    for consumer in consumers:
        consumer.start()

    # Enqueue jobs
    num_jobs = 10
    for _ in xrange(num_jobs):
        tasks.put(Task())

    # Add a poison pill for each consumer
    for _ in xrange(num_consumers):
        tasks.put(None)

    # Wait for all tasks to finish
    tasks.join()

    # Start printing results
    while num_jobs:
        result = results.get()
        print 'Result:', result
        num_jobs -= 1

将列表中的对象删除为传递给多处理

1 个答案: