我有一个多线程函数,我想要一个使用tqdm
的状态栏。是否有一种简单的方法来显示带有ThreadPoolExecutor
的状态栏?正是并行化部分使我感到困惑。
import concurrent.futures
def f(x):
return f**2
my_iter = range(1000000)
def run(f,my_iter):
with concurrent.futures.ThreadPoolExecutor() as executor:
function = list(executor.map(f, my_iter))
return results
run(f, my_iter) # wrap tqdr around this function?
答案 0 :(得分:11)
您可以将tqdm
包裹在executor
周围,如下所示以跟踪进度:
list(tqdm(executor.map(f, iter), total=len(iter))
这是您的示例:
import time
import concurrent.futures
from tqdm import tqdm
def f(x):
time.sleep(0.001) # to visualize the progress
return x**2
def run(f, my_iter):
with concurrent.futures.ThreadPoolExecutor() as executor:
results = list(tqdm(executor.map(f, my_iter), total=len(my_iter)))
return results
my_iter = range(100000)
run(f, my_iter)
结果是这样的:
16%|██▏ | 15707/100000 [00:00<00:02, 31312.54it/s]
答案 1 :(得分:8)
极度赞成和接受的答案的问题在于,ThreadPoolExecutor.map
函数必须生成结果的顺序不是可用的。因此,如果myfunc
的第一次调用恰好是例如要完成的最后一次调用,则进度条将同时从0%变为100%,并且仅在所有调用完成时才进入。最好将ThreadPoolExecutor.submit
与as_completed
一起使用:
import time
import concurrent.futures
from tqdm import tqdm
def f(x):
time.sleep(0.001) # to visualize the progress
return x**2
def run(f, my_iter):
l = len(my_iter)
with tqdm(total=l) as pbar:
# let's give it some more threads:
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
futures = {executor.submit(f, arg): arg for arg in my_iter}
results = {}
for future in concurrent.futures.as_completed(futures):
arg = futures[future]
results[arg] = future.result()
pbar.update(1)
print(321, results[321])
my_iter = range(100000)
run(f, my_iter)
打印:
321 103041
这只是一般的想法。根据{{1}}的类型,可能无法直接将my_iter
函数直接应用到它,而无需先将其转换为列表。要点是将len
与submit
一起使用。
答案 2 :(得分:-1)
我认为最简短的方法是:
with ThreadPoolExecutor(max_workers=20) as executor:
results = list(tqdm(executor.map(myfunc, range(len(my_array))), total=len(my_array)))