Question

我正在尝试使用Python的多处理模块来加速处理。但是，当我在代码的最底部运行test_parallel_compute函数时，在具有32个节点的计算集群上运行（编辑我发现我只在一个节点上运行节点），程序在没有多处理的情况下运行的时间更长：1024秒（32个进程）对231秒（不使用多处理模块）。这些秒的1022秒花费在parallel_compute_new_2函数中的pool.map调用中，因此时间不受分区输入或加入返回函数的限制。

我有一个输入列表（b）和函数的几个其他参数（a和c）（test_function）。为了为多个处理器准备这些，我分区b。然后我将函数及其分区参数作为参数提供给worker_function_new，它在其分区参数上调用test_function。

问题编辑：您是否可以看到映射多个流程的效率低下如下？同样，在parallel_compute_new_2函数内的pool.map调用中花费了1022秒，因此时间不受分区输入和加入返回函数的限制。

我正在使用a = 100.0，b =范围（10000000）和c = 15.0的输入来调用它。

谢谢！

# Partition the input list
def partition_inputs(input, number):
    num_inputs = len(input)
    return [input[num_inputs * i/number:num_inputs * (i+1)/number] for i in range(number)]

# This is the function that each process is supposed to run.
# It takes in several arguments. b is a long list, which is partitioned
# into multiple slices for each process. a and c are just other numbers.
# This function's return values, d, e, and f, are joined between each process.
def test_function_2(args):
    a = args[0]
    b = args[1]
    c = args[2]
    d = []
    e = 0
    f = {}
    for i in b:
        d.append(a*i*c)
        f[i] = set([a, i, c])
    return d, e, f



def parallel_compute_new_2(function, args, input, input_index, partition_input_function, join_functions_dict, pool,
                     number=32, procnumber=32):
    # Partition the b list. In my case, the partition_input_function is
    # partition_input_list, as above.

    new_inputs = partition_input_function(input, number)

    # Since test_function_2 requires arguments (a, c) beyond the partitioned
    # list b, create a list of the complete arguments. 
    worker_function_args_list = []
    for i in range(number):
        new_args = args[:]
        new_args[input_index] = new_inputs[i]
        worker_function_args_list.append(new_args)

    returnlist = pool.map(function, worker_function_args_list)

    # Join the return values from each process.
    return_values = list(returnlist[0])

    for index in join_functions_dict:
        for proc in range(1, number):
            return_values[index] = join_functions_dict[index](return_values[index], returnlist[proc][index])

    return return_values

def test_parallel_compute(a, b, c, number=32, procnumber=32):


    join_functions_dict = {}
    join_functions_dict[0] = lambda a, b: a + b
    join_functions_dict[2] = combine_dictionaries

    # a = 100.
    # b = range(1000000000)
    # c = 15.

    d, e, f = test_function(a, b, c)

    pool = mup.Pool(processes=procnumber)

    d1, e1, f1 = parallel_compute_new_2(test_function_2, [a, b, c], b, 1, partition_inputs, join_functions_dict, pool, number=number, procnumber=procnumber)

多处理模块运行较慢的工作功能？

0 个答案: