Question

我正在尝试并行化带有多个常量参数的函数。到目前为止，我已经能够运行它，但是它没有使过程并行化。我应该如何处理？

我尝试执行以下操作：

import numpy as np
import multiprocessing 


def optm(hstep,astep,time_ERA):

    #this is a secondary function where I get arrays from a dataset
    data = checkdate(time_ERA,2,4)
    zlevels=data[0] 
    pottemp=data[1] 


    for z1 in np.linspace(0,zlevels[-1],hstep):
        for z2 in np.linspace(0,zlevels[-1],hstep):
            for a1 in np.linspace(0,0.01,astep): # max angle
                for a2 in np.linspace(0,0.01,astep):
                    for a3 in np.linspace(0,0.01,astep):
                       result_array=another_function(zlevels,pottemp,z1,z2,a1,a2,a3) # this function is the one that does all the math in the code. Therefore, it take a lot of time to compute it.

    return result_array

然后我以这种方式并行化该函数：

input_list = [(hstep,astep,time_ERA)] #creat a tuple for the necessary startmap 

pool = multiprocessing.Pool()
result = pool.starmap(optm, input_list)
pool.close()

当我运行它时，它比没有并行化要花费更长的时间。这是我第一次尝试并行化代码，因此我仍然不确定应该使用map还是starmap以及如何对其进行并行化。

Answer 1

使用我的评论中的最小示例适合您的问题：

import multiprocessing
import time
import numpy as np

def optm(hstep,astep,time_ERA):
    values = []
    #this is a secondary function where I get arrays from a dataset
    data = checkdate(time_ERA,2,4)
    zlevels=data[0] 
    pottemp=data[1] 
    for z1 in np.linspace(0,zlevels[-1],hstep):
        for z2 in np.linspace(0,zlevels[-1],hstep):
            for a1 in np.linspace(0,0.01,astep): # max angle
                for a2 in np.linspace(0,0.01,astep):
                    for a3 in np.linspace(0,0.01,astep):
                        values.append([zlevels,pottemp,z1,z2,a1,a2,a3])
    return values

def mp_worker((zlevels,pottempt,z1,z2,a1,a2,a3)):
    temp = another_function(zlevels,pottemp,z1,z2,a1,a2,a3)
    # do something with the result

def mp_handler(data):
    p = multiprocessing.Pool(2) # Change 2 to your cpu count
    p.map(mp_worker, data)

if __name__ == '__main__':
    data = optm(hstep,astep,time_ERA) 
    mp_handler(data)

您可以对每组参数执行pool.apply_async()而不是映射，也可以使用多处理队列将作业提供给子流程。我假设输出需要存储在一个单独的数组中，因此使用Queue可以使此操作变得更加容易。您可以将作业送入队列，然后将结果推入另一个队列，当所有进程完成后，从主线程中的结果队列中收集结果并将其存储到数组中。

如何并行化需要多个常量的函数？

1 个答案: