Question

亲爱的python并行大师，

我设计了一种基于serial_worker函数的并行化方案：

import pathos
import multiprocessing
pool = pathos.pools.ProcessPool(ncpus=n_cores)
amps=[None,] * len(tr)  # tr is a long array
for i, _amps in enumerate(pool.map(serial_worker, tr)):
    amps[i] = _amps
pool.terminate()

我知道在调用函数时python会有很大的开销。在这里，我给serial_worker打电话了len(tr)次。我可以取消该功能吗？以下是完整的问题，展示了我最终如何使用serial_worker函数：

import numpy as np
import pathos  # enables pickling serial_worker
import multiprocessing

# bi and q are modest size arrays
bi = np.random.rand(42)
q = np.random.rand(10, 3)
tr = np.random.rand(10000, 42, 3)  # first dimension is very looong

def serial_worker(frame):
    """frame.shape = (42, 3)
    Uses arrays q and bi defined previously, a sort of closure 
    """
    exponents = np.tensordot(frame, q, axes=(1, 1))  # shape=(42, 10)
    exponentials = np.exp(1j * exponents)
    return np.tensordot(bi, exponentials, axes=1)  # shape=(10,)

pool = pathos.pools.ProcessPool(ncpus=n_cores)
amps=[None,] * len(tr)  # tr is a long array
for i, _amps in enumerate(pool.map(serial_worker, tr)):
    amps[i] = _amps
pool.terminate()

我将q和bi隐藏在serial_worker中，以每次调用一个函数为代价。谁能建议其他并行化方案？

谢谢

并行化替代了对函数的并行调用？

0 个答案: