Question

我生成100个大小为1000x1000的随机整数矩阵。我使用多处理模块来计算100个矩阵的特征值。

代码如下：

import timeit
import numpy as np
import multiprocessing as mp

def calEigen():

 S, U = np.linalg.eigh(a)

def multiprocess(processes):
 pool = mp.Pool(processes=processes)
 #Start timing here as I don't want to include time taken to initialize the processes
 start = timeit.default_timer()
 results = [pool.apply_async(calEigen, args=())]
 stop = timeit.default_timer()
 print (processes":", stop - start) 

 results = [p.get() for p in results]
 results.sort() # to sort the results 


if __name__ == "__main__":

 global a
 a=[]

 for i in range(0,100):
  a.append(np.random.randint(1,100,size=(1000,1000)))

 #Print execution time without multiprocessing
 start = timeit.default_timer()
 calEigen()
 stop = timeit.default_timer()
 print stop - start 

 #With 1 process
 multiprocess(1)

 #With 2 processes
 multiprocess(2)

 #With 3 processes
 multiprocess(3)

 #With 4 processes
 multiprocess(4)

输出

0.510247945786
('Process:', 1, 5.1021575927734375e-05)
('Process:', 2, 5.698204040527344e-05)
('Process:', 3, 8.320808410644531e-05)
('Process:', 4, 7.200241088867188e-05)

另一次迭代显示了这个输出：

 69.7296020985
 ('Process:', 1, 0.0009050369262695312)
 ('Process:', 2, 0.023727893829345703)
 ('Process:', 3, 0.0003509521484375)
 ('Process:', 4, 0.057518959045410156)

我的问题是这些：

为什么时间执行时间不会减少流程增加？我是否正确使用多处理模块？
我是否正确计算执行时间？

我编辑了下面评论中给出的代码。我希望串行和多处理函数能够找到100个矩阵的相同列表的特征值。编辑的代码是 -

import numpy as np
import time
from multiprocessing import Pool

a=[]

for i in range(0,100):
 a.append(np.random.randint(1,100,size=(1000,1000)))

def serial(z):
 result = []
 start_time = time.time()
 for i in range(0,100):    
  result.append(np.linalg.eigh(z[i])) #calculate eigen values and append to result list
 end_time = time.time()
 print("Single process took :", end_time - start_time, "seconds")


def caleigen(c):  
 result = []        
 result.append(np.linalg.eigh(c)) #calculate eigenvalues and append to result list
 return result

def mp(x):
 start_time = time.time()
 with Pool(processes=x) as pool:  # start a pool of 4 workers
  result = pool.map_async(caleigen,a)   # distribute work to workers
  result = result.get() # collect result from MapResult object
 end_time = time.time()
 print("Mutltiprocessing took:", end_time - start_time, "seconds" )

if __name__ == "__main__":

 serial(a)
 mp(1,a)
 mp(2,a)
 mp(3,a)
 mp(4,a)

随着进程数量的增加，时间没有减少。我哪里错了？多处理是否将列表划分为进程的块或者我必须进行除法？

Answer 1

您没有正确使用多处理模块。正如@dopstar指出的那样，你没有分开你的任务。流程池只有一个任务，所以无论你分配了多少工人，只有一个人能得到这份工作。至于你的第二个问题，我没有使用timeit来精确测量处理时间。我只是使用time模块来粗略地了解事物的速度。不过，它大部分时间都是为了达到目的。如果我理解您正在尝试正确执行的操作，那么这应该是代码的单进程版本

import numpy as np
import time

result = []
start_time = time.time()
for i in range(100):
    a = np.random.randint(1, 100, size=(1000,1000))  #generate random matrix
    result.append(np.linalg.eigh(a))                 #calculate eigen values and append to result list
end_time = time.time()
print("Single process took :", end_time - start_time, "seconds")

单个进程版本在我的计算机上耗时15.27秒。下面是多进程版本，我的计算机只用了0.46秒。我还包括单个过程版本进行比较。（单个进程版本也必须包含在if块中，并放在多进程版本之后。）因为您想重复计算100次，所以创建池要容易得多工人们让他们自动承担未完成的任务，而不是手动启动每个流程并指定每个流程应该做什么。在我的代码中，caleigen调用的参数仅仅是跟踪任务执行的次数。最后，map_async通常比apply_async更快，其缺点是消耗稍多的内存并且只为函数调用使用一个参数。使用map_async但不使用map的原因是，在这种情况下，返回结果的顺序无关紧要，map_async比map快得多。

from multiprocessing import Pool
import numpy as np
import time

def caleigen(x):     # define work for each worker
    a = np.random.randint(1,100,size=(1000,1000))   
    S, U = np.linalg.eigh(a)                        
    return S, U


if __name__ == "main":
    start_time = time.time()
    with Pool(processes=4) as pool:      # start a pool of 4 workers
        result = pool.map_async(caleigen, range(100))   # distribute work to workers
        result = result.get()        # collect result from MapResult object
    end_time = time.time()
    print("Mutltiprocessing took:", end_time - start_time, "seconds" )

    # Run the single process version for comparison. This has to be within the if block as well. 
    result = []
    start_time = time.time()
    for i in range(100):
        a = np.random.randint(1, 100, size=(1000,1000))  #generate random matrix
        result.append(np.linalg.eigh(a))                 #calculate eigen values and append to result list
    end_time = time.time()
    print("Single process took :", end_time - start_time, "seconds")

用于计算特征值的多处理

1 个答案: