使用tqdm与current.futures?

时间:2018-07-30 20:21:40

标签: python concurrent.futures tqdm

我有一个多线程函数,我想要一个使用tqdm的状态栏。是否有一种简单的方法来显示带有ThreadPoolExecutor的状态栏?正是并行化部分使我感到困惑。

import concurrent.futures

def f(x):
    return f**2

my_iter = range(1000000)

def run(f,my_iter):
    with concurrent.futures.ThreadPoolExecutor() as executor:
        function = list(executor.map(f, my_iter))
    return results

run(f, my_iter) # wrap tqdr around this function?

3 个答案:

答案 0 :(得分:11)

您可以将tqdm包裹在executor周围,如下所示以跟踪进度:

list(tqdm(executor.map(f, iter), total=len(iter))

这是您的示例:

import time  
import concurrent.futures
from tqdm import tqdm

def f(x):
    time.sleep(0.001)  # to visualize the progress
    return x**2

def run(f, my_iter):
    with concurrent.futures.ThreadPoolExecutor() as executor:
        results = list(tqdm(executor.map(f, my_iter), total=len(my_iter)))
    return results

my_iter = range(100000)
run(f, my_iter)

结果是这样的:

16%|██▏           | 15707/100000 [00:00<00:02, 31312.54it/s]

答案 1 :(得分:8)

极度赞成和接受的答案的问题在于,ThreadPoolExecutor.map函数必须生成结果的顺序不是可用的。因此,如果myfunc的第一次调用恰好是例如要完成的最后一次调用,则进度条将同时从0%变为100%,并且仅在所有调用完成时才进入。最好将ThreadPoolExecutor.submitas_completed一起使用:

import time
import concurrent.futures
from tqdm import tqdm

def f(x):
    time.sleep(0.001)  # to visualize the progress
    return x**2

def run(f, my_iter):
    l = len(my_iter)
    with tqdm(total=l) as pbar:
        # let's give it some more threads:
        with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
            futures = {executor.submit(f, arg): arg for arg in my_iter}
            results = {}
            for future in concurrent.futures.as_completed(futures):
                arg = futures[future]
                results[arg] = future.result()
                pbar.update(1)
    print(321, results[321])

my_iter = range(100000)
run(f, my_iter)

打印:

321 103041

这只是一般的想法。根据{{​​1}}的类型,可能无法直接将my_iter函数直接应用到它,而无需先将其转换为列表。要点是将lensubmit一起使用。

答案 2 :(得分:-1)

我认为最简短的方法是:

with ThreadPoolExecutor(max_workers=20) as executor:
    results = list(tqdm(executor.map(myfunc, range(len(my_array))), total=len(my_array)))