Question

我有一个主要方法可以生成多个流程（在交叉验证中训练k折），因为我想同时训练所有折。

像这样

def main(model=None, input_dir=None, output_dir=None, n_iter=100, dropout=0.5, batchsize=32.0):

        inputpath = os.path.abspath(input_dir)
        outputpath = os.path.abspath(output_dir)

        path = Path(inputpath)
        onlyfiles = [e for e in path.iterdir() if e.is_file()]

        for i in onlyfiles:
                if i.name.startswith("TRAIN_"):
                        p = multiprocessing.Process(target=worker, args=(model, i, outputpath, n_iter, dropout, batchsize))
                        p.start()

然后最后我要执行的是运行一个评估结果的函数。但是我认为，一旦所有进程启动，主要功能就会退出。

只有完成所有过程并创建所有模型后，我才能进行评估。我该怎么办？

Answer 1

您需要将它们全部收集起来，然后对所有的它们调用join()：

def main(model=None, input_dir=None, output_dir=None, n_iter=100, dropout=0.5, batchsize=32.0):
    # stuff elided ...
    all_processes = []
    for i in onlyfiles:
        if i.name.startswith("TRAIN_"):
            p = multiprocessing.Process(target=worker, args=(model, i, outputpath, n_iter, dropout, batchsize))
            p.start()
            all_processes.append(p)
    for p in all_processes:
        p.join()
    # Now do something else

Answer 2

您可以尝试使用带有回调的pool.starmap_async()方法：

import multiprocessing

from multiprocessing import Pool

def callback(results):
    print(f'Processing: {results}')


def main(model=None, input_dir=None, output_dir=None, n_iter=100, dropout=0.5, batchsize=32.0):
    # stuff elided ...
    cpus = multiprocessing.cpu_count()
    args = ((model, i, outputpath, n_iter, dropout, batchsize) for i in onlyfiles)
    with Pool(processes=cpus) as pool:
        results = pool.starmap_async(worker, args, callback=callback)
        print(f'Done, results: {results.get()}')

或简单地：

results = pool.starmap(worker, args)
print(f'Processing: {results}')

我如何等待所有过程完成然后做什么？

2 个答案: