一次使用两个线程池

Question

我只是想用Python来了解ThreadPool，我想着一种情况，其中一个线程池产生数据而另一个线程池消耗数据。

以下是我对此的幼稚尝试：

from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed
from time import sleep
import threading
import random

final_results = {}

def sample_function(n):
    threadname = threading.current_thread().name
    print(f"Starting execution of thread with the name {threadname} and argument {n}")
    sleep(random.randint(1, 5))
    print(f"About to finish executing for thread name {threadname}")
    return n ** 2

def sample_function2(number, lock):
    threadname = threading.current_thread().name
    print(f"Starting execution of thread from 2nd pool with the name {threadname} and working on number {number}")
    with lock:
        final_results[number] = number * 2
    print(f"Completing execution of thread with the name {threadname}")


pool2 = ThreadPoolExecutor(max_workers=3)
numbers1 = [1, 2, 3, 4, 5, 6]
with ThreadPoolExecutor(max_workers=3) as executor:
    lock = threading.Lock()
    futures = []
    for result in executor.map(sample_function, numbers1):
        future = pool2.submit(sample_function2, (result, lock))
        futures.append(future)

for future in futures:
    while not future.done():
        sleep(2)

print("Everything is done ... checking results now !")
print(final_results)

这是我得到的输出：

Starting execution of thread with the name ThreadPoolExecutor-1_0 and argument 1
Starting execution of thread with the name ThreadPoolExecutor-1_1 and argument 2
Starting execution of thread with the name ThreadPoolExecutor-1_2 and argument 3
About to finish executing for thread name ThreadPoolExecutor-1_2
About to finish executing for thread name ThreadPoolExecutor-1_1
Starting execution of thread with the name ThreadPoolExecutor-1_2 and argument 4
Starting execution of thread with the name ThreadPoolExecutor-1_1 and argument 5
About to finish executing for thread name ThreadPoolExecutor-1_0
Starting execution of thread with the name ThreadPoolExecutor-1_0 and argument 6
About to finish executing for thread name ThreadPoolExecutor-1_1
About to finish executing for thread name ThreadPoolExecutor-1_2
About to finish executing for thread name ThreadPoolExecutor-1_0
Everything is done ... checking results now !
{}

那么为什么第二个池中的任何线程都没有启动？我要去哪里错了，因为我也看不到任何错误消息（也请指导我为什么）。

这是解决问题的正确方法，还是线程池使用某种队列之间进行通信？

但是无论哪种情况，我都必须使第二个池起作用。

Answer 1

以下是对您的问题的至少部分答案，它修复了第二个池未“启动”的问题。造成这种情况的原因有很多。

其中两个是您没有正确调用pool2.submit()或从中获取结果（您需要附加从 calling future.result返回的值）。

由于futures不会返回任何内容，因此None列表中的值全部为sample_function2()，因此while not future.done():无效，因为每个元素都是None并且没有done方法可以调用，因此我删除了整个for future in futures:循环-完全没有必要，因为到那时所有提交的线程都已经完成了先前的循环已写入（即调用submit()，然后立即通过调用result()等待其结束）。

这意味着使用两个Pool的全部两种方法不会做那么多的并发处理，因此无法充分利用该功能。就是说，我真的不明白为什么您会觉得必须拥有它们-因此很难知道建议您做什么。

from concurrent.futures import ThreadPoolExecutor
from time import sleep
import threading
import random

final_results = {}

def sample_function(n):
    threadname = threading.current_thread().name
    print(f"Starting execution of thread with the name {threadname} and argument {n}")
    sleep(random.randint(1, 5))
    print(f"About to finish executing for thread name {threadname}")
    return n ** 2

def sample_function2(number, lock):
    threadname = threading.current_thread().name
    print(f"Starting execution of thread from 2nd pool with the name {threadname} and working on number {number}")
    with lock:
        final_results[number] = number * 2
    print(f"Completing execution of thread with the name {threadname}")


numbers1 = [1, 2, 3, 4, 5, 6]

pool2 = ThreadPoolExecutor(max_workers=3)
with ThreadPoolExecutor(max_workers=3) as executor:
    lock = threading.Lock()
    futures = []
    for result in executor.map(sample_function, numbers1):
        future = pool2.submit(sample_function2, result, lock)  # Pass args like this.
        futures.append(future.result())  # Must call result(), but value always None.

print("Everything is done ... checking results now!")
print(final_results)

输出：

Starting execution of thread with the name ThreadPoolExecutor-1_0 and argument 1
Starting execution of thread with the name ThreadPoolExecutor-1_1 and argument 2
Starting execution of thread with the name ThreadPoolExecutor-1_2 and argument 3
About to finish executing for thread name ThreadPoolExecutor-1_1
Starting execution of thread with the name ThreadPoolExecutor-1_1 and argument 4
About to finish executing for thread name ThreadPoolExecutor-1_2
Starting execution of thread with the name ThreadPoolExecutor-1_2 and argument 5
About to finish executing for thread name ThreadPoolExecutor-1_0
Starting execution of thread with the name ThreadPoolExecutor-1_0 and argument 6
Starting execution of thread from 2nd pool with the name ThreadPoolExecutor-0_0 and working on number 1
Completing execution of thread with the name ThreadPoolExecutor-0_0
Starting execution of thread from 2nd pool with the name ThreadPoolExecutor-0_0 and working on number 4
Completing execution of thread with the name ThreadPoolExecutor-0_0
Starting execution of thread from 2nd pool with the name ThreadPoolExecutor-0_0 and working on number 9
Completing execution of thread with the name ThreadPoolExecutor-0_0
About to finish executing for thread name ThreadPoolExecutor-1_2
About to finish executing for thread name ThreadPoolExecutor-1_1
Starting execution of thread from 2nd pool with the name ThreadPoolExecutor-0_0 and working on number 16
Completing execution of thread with the name ThreadPoolExecutor-0_0
Starting execution of thread from 2nd pool with the name ThreadPoolExecutor-0_1 and working on number 25
Completing execution of thread with the name ThreadPoolExecutor-0_1
About to finish executing for thread name ThreadPoolExecutor-1_0
Starting execution of thread from 2nd pool with the name ThreadPoolExecutor-0_0 and working on number 36
Completing execution of thread with the name ThreadPoolExecutor-0_0
Everything is done ... checking results now!
{1: 2, 4: 8, 9: 18, 16: 32, 25: 50, 36: 72}