Question

我想对 ThreadPool 内正在处理的大量项目实施节流功能，方法是仅将 250 项目限制在一段时间。

pool = ThreadPool(250)

def func(i):
    print(i)
    time.sleep(3)

i = 0 
while True: # The amount of items is virtually infinite
    i+=1
    # Block here - if there are no free threads available in the pool
    pool.map_async(func,[i]) # Once delegated, don't block and move forward

如何实现评论中说明的功能？

Answer 1

您可以使用池中的线程数初始化 threading.Semaphore，然后使用此信号量初始化线程池本身。然后每个工作人员将通过在信号量上发出对 release 的调用来表示它已完成其任务并释放正在运行它的线程，这会增加信号量计数。因此，信号量维护空闲线程数的计数。在向池提交新任务之前，在信号量上调用 acquire。如果信号量计数不为正（至少有一个空闲线程），这将阻塞。否则，信号量计数递减（现在空闲线程减少了一个）。

from multiprocessing.pool import ThreadPool
from threading import Semaphore
import time


def func(i):
    print(i)
    time.sleep(3)
    semaphore.release() # show this thread is now free

POOLSIZE = 10

semaphore = Semaphore(value=POOLSIZE)
pool = ThreadPool(POOLSIZE)
i = 0
for i in range(20):
    # Block here - if there are no free threads available in the pool
    semaphore.acquire()
    """
    # to demonstrate that we are blocking, instead of doing a simple semaphore.aquire():
    if semaphore.acquire(blocking=False) is False:
        print('We would block for i =', i)
        semaphore.acquire() # do actual blocking call
    """
    pool.apply_async(func, args=(i,)) # Once delegated, don't block and move forward
# wait for all tasks to complete:
pool.close()
pool.join()

使用回调

from multiprocessing.pool import ThreadPool
from threading import Semaphore
import time


def func(i):
    print(i)
    time.sleep(3)

POOLSIZE = 10

def my_callback(result):
    semaphore.release() # show this thread is now free


semaphore = Semaphore(value=POOLSIZE)
pool = ThreadPool(POOLSIZE)
i = 0
for i in range(20):
    # Block here - if there are no free threads available in the pool
    semaphore.acquire()
    """
    # to demonstrate that we are blocking, instead of doing a simple semaphore.aquire():
    if semaphore.acquire(blocking=False) is False:
        print('We would block for i =', i)
        semaphore.acquire() # do actual blocking call
    """
    pool.apply_async(func, args=(i,), callback=my_callback) # Once delegated, don't block and move forward
    # or:
    #pool.apply_async(func, args=(i,), callback=lambda result: semaphore.release()) # Once delegated, don't block and move forward

# wait for all tasks to complete:
pool.close()
pool.join()

ThreadPool 等待直到线程在池中可用（节流）

1 个答案: