Question

我想利用python的多处理模块来并行化这个简单的例子：

import numpy as np
import h5py
import os
import matplotlib.pyplot as plt
from multiprocessing import Pool

def load_array(path, variable):
    try:
        return np.array(h5py.File(path, "r").get(variable))
    except:
        raise FileNotFoundError("Corrupted file: {}".format(path))

def mat2img(rootdir, save_path, variable):

    fig = plt.figure()

    print("Processing " + rootdir)

    for subdir, dirs, files in os.walk(rootdir):
        for file in files:
            arr = load_array(os.path.join(subdir, file), variable).T

            fig.subplots_adjust(top=1, bottom=0, right=1, left=0)
            plt.pcolormesh(np.arange(0, arr.shape[1]), np.arange(0, arr.shape[0]), arr, cmap="jet")
            plt.axis("off")
            plt.savefig(os.path.join(save_path, subdir.split(os.path.sep)[-1], file + ".jpg"))
            plt.clf()

if __name__ == '__main__':

    with Pool(processes=3) as pool:
        pool.apply_async(mat2img, ("O:\\data1", "O:\\spectrograms", "spectrum"))
        pool.apply_async(mat2img, ("O:\\data2", "O:\\spectrograms", "spectrum"))
        pool.apply_async(mat2img, ("O:\\data3", "O:\\spectrograms", "spectrum"))

然而，如果apply_async没有调用任何函数，这没有任何作用。从documentation我看到每个apply_async都分配给某个变量res。即使我的功能没有返回任何内容，我是否也需要这样做？如果是这样，那个变量res包含什么，我将调用get()？我在哪里弄错了？

Answer 1

您使用appy_async安排作业。然后你必须等到他们完成。如果你不等，他们甚至都不会开始。

with Pool(processes=3) as pool:
    pool.apply_async(mat2img, ("O:\\data1", "O:\\spectrograms", "spectrum"))
    pool.apply_async(mat2img, ("O:\\data2", "O:\\spectrograms", "spectrum"))
    pool.apply_async(mat2img, ("O:\\data3", "O:\\spectrograms", "spectrum"))
    pool.close()  # Do not accept any more jobs.
    pool.join(timeout=1000)  # Wait until all async jobs complete.

或者，您可以.get()确保每项工作完成：

with Pool(processes=3) as pool:
    # Schedule the jobs.
    jobs = [pool.apply_async(mat2img, (dest, "O:\\spectrograms", "spectrum"))
            for dest in ("O:\\data1", "O:\\data2", "O:\\data3")]
    # Wait for the jobs to complete.
    for job in jobs:
        job.get(timeout=100)

正如@AndreaCorbellini正确指出的那样，如果你的工作没有返回你关心的任何结果，你可以job.wait()而不是job.get()。

如果我在python中使用多处理但是我的函数不返回任何内容，我是否需要调用get（）？

1 个答案: