Question

我在下面有这个代码，我已经优化了算法，使其尽可能快，但它仍然太慢。所以a正在考虑使用多处理（我对这种东西没有任何影响），但是我尝试了一些使用池和线程的东西，但要么比以前慢，要么没有工作。因此，我想知道我应该如何做到这一点，以便它能够运作并且速度更快。如果除了多线程之外还有其他选项可以更快地制作代码。

def calc(indices, data):
    matrix = [[0] * len(indices) for i in range(len(indices))]
    for i_a, i_b in list(itertools.combinations(indices, 2)):
        a_res, b_res = algorithm(data[i_a], data[i_b])
       matrix[i_b][i_a] = a_res
       matrix[i_a][i_b] = b_res
    return matrix


def algorithm(a,b):
   # Verry slow and complex

Answer 1

基于 Simon 的回答，这是一个将multiprocessing池应用于您的问题版本的示例。您的里程将根据您的机器上有多少核心而有所不同，但我希望这将有助于演示如何为您的问题构建解决方案：

import itertools
import numpy as np
import multiprocessing as mp
import time

def calc_mp(indices, data):
    # construct pool
    pool = mp.Pool(mp.cpu_count())

    # we are going to populate the matrix; organize all the inputs; then map them
    matrix = [[0] * len(indices) for i in range(len(indices))]
    args = [(data[i_a], data[i_b]) for i_a, i_b in list(itertools.combinations(indices, 2))]
    results = pool.starmap(algorithm, args)

    # unpack the results into the matrix
    for i_tuple, result in zip([(i_a, i_b) for i_a, i_b in list(itertools.combinations(indices, 2))], results):
        # unpack
        i_a, i_b = i_tuple
        a_res, b_res = result

        # set it in the matrix
        matrix[i_b][i_a] = a_res
        matrix[i_a][i_b] = b_res

    return matrix

def calc_single(indices, data):
    # do the simple single process version
    matrix = [[0] * len(indices) for i in range(len(indices))]
    for i_a, i_b in list(itertools.combinations(indices, 2)):
        a_res, b_res = algorithm(data[i_a], data[i_b])
        matrix[i_b][i_a] = a_res
        matrix[i_a][i_b] = b_res

    return matrix

def algorithm(a,b):
    # Very slow and complex
    time.sleep(2)
    return a + b, a - b

if __name__ == "__main__":
    # generate test data;
    indices = range(5)
    data = range(len(indices))

    # test single
    time_start = time.time()
    print(calc_single(indices, data))
    print("Took {}".format(time.time() - time_start))

    # mp
    time_start = time.time()
    print(calc_mp(indices, data))
    print("Took {}".format(time.time() - time_start))

结果，有8个核心，

[[0, -1, -2, -3, -4], [1, 0, -1, -2, -3], [2, 3, 0, -1, -2], [3, 4, 5, 0, -1], [4, 5, 6, 7, 0]]
Took 20.02155065536499
[[0, -1, -2, -3, -4], [1, 0, -1, -2, -3], [2, 3, 0, -1, -2], [3, 4, 5, 0, -1], [4, 5, 6, 7, 0]]
Took 4.073369264602661

Answer 2

Multiprocessing中你最好的选择。您需要将数据分区为块并将每个块传递给进程。线程无法帮助您使用Python，因为所有Python进程都在单个cpu线程上运行。 It's still useful for some use cases，例如您可能会阻止其中一些活动的地方，而不是并行工作负载。

Python多线程/进程用值填充矩阵

2 个答案: