Question

我需要将一个函数并行应用于大型生成器中的所有项目。我不想缓冲返回，它只是一个巨大的None数组。

我不明白如何使用concurrent.future API。

通过带有工人上限的线程池，将函数应用于生成器中的项目，放弃结果的最简单，最有效的方法是什么？

Answer 1

好的，这就是我的所作所为。鉴于以下定义：

from concurrent import futures
import itertools

func    = print          # any function
tasks   = iter(iterable) # any one-time-only iterator
workers = 10

这会将func映射到tasks，并带有工人上限，会丢弃结果：

with futures.ThreadPoolExecutor(max_workers = workers) as pool:
      initial = itertools.islice(tasks, workers)
      running = set(pool.submit(func, task) for task in initial)

      for task in tasks:
          done, _  = futures.wait(running, return_when = futures.FIRST_COMPLETED)
          running -= done
          running.add(pool.submit(func, task))

Answer 2

in case you wish to extend the implementation for multiple objects whom iterations are like state-machine you can use a program that I have written.

https://github.com/talvezu/Python/blob/master/examples/concurrency/producer_consumer_generatored_thread_poll_with_inheritance.py

2 contexts. one reading tasks from file (2 different objects representing 2 state-machines) one context hold the thread-pool and any time a future is done it is using a thread-safe queue to notify that it is done, which causes the next() method to be invoked and continue the next iteration.

ones the object reaching "done" state it is no longer returned to the pool.

Python3线程池：将函数应用于任务生成器

2 个答案: