Question

我有这个通用功能，我用它来运行许多不同的操作，这些操作可以启动多个进程，然后使用progressbar2模块跟踪进度。有时，进度条会导致挂断并导致功能卡住。我似乎无法弄清楚为什么，如果我删除进度条更新，然后该过程完成并继续前进，所以问题似乎是在while循环中。有没有更好的方法来构建while循环？

def runMultiProgress(function, inputList, cpus):
    from progressbar import ProgressBar, Percentage, ETA
    from time import sleep
    #setup pool
    p = multiprocessing.Pool(cpus)
    #setup progress bar
    widgets = ['     Progress: ', Percentage(),' || ', ETA()]
    pbar = ProgressBar(widgets=widgets, term_width=30, maxval=len(inputList)).start()
    #setup results and split over cpus
    results = []
    r = [p.apply_async(function, (x,), callback=results.append) for x in inputList]
    #refresh pbar every 5 seconds
    while len(results) != len(inputList):
        pbar.update(len(results))
        sleep(5)
    pbar.finish()
    p.close()
    p.join()

更新：引用我的消息来源，这个多处理+进度条来自@ julien-tourille，在这里回答：Show the progress of a Python multiprocessing pool map call?

Answer 1

我最终修改了函数以仅使用简单的进度条并修改了如何调用多处理。这似乎更简单，对我有用。

def runMultiProgress(function, inputList, cpus):
    #setup pool
    p = multiprocessing.Pool(cpus)
    #setup results and split over cpus
    tasks = len(inputList)
    results = []
    for i in inputList:
        results.append(p.apply_async(function, [i]))
    #refresh progress every sec
    while True:
        incomplete_count = sum(1 for x in results if not x.ready())
        if incomplete_count == 0:
            break
        sys.stdout.write("Progress: %.2f%% \r" % (float(tasks - incomplete_count) / tasks * 100))
        sys.stdout.flush()
        time.sleep(1)
    p.close()
    p.join()

Python多处理和进度条功能偶尔会挂起

1 个答案: