Question

我已经浏览了一段时间，但找不到任何我能理解的建设性答案。

我应该如何对以下代码进行并列化：

import random
import math
import numpy as np
import sys
import multiprocessing

boot = 20#number of iterations to be performed
def myscript(iteration_number):  
    #stuff that the code actually does


def main(unused_command_line_args):
    for i in xrange(boot):
        myscript(i)
    return 0

if __name__ == '__main__':
    sys.exit(main(sys.argv))

或者我在哪里可以阅读它？我甚至不确定如何搜索它。

Answer 1

对于一批令人尴尬的并行工作而言，从for循环到并行的自然发展是非常自然的。

>>> import multiprocess as mp
>>> # build a target function
>>> def doit(x):
...   return x**2 - 1
... 
>>> x = range(10)
>>> # the for loop
>>> y = []   
>>> for i in x:
...   y.append(doit(i))
... 
>>> y
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]

那么如何并行处理这个功能呢？

>>> # convert the for loop to a map (still serial)
>>> y = map(doit, x)
>>> y
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]
>>> 
>>> # build a worker pool for parallel tasks
>>> p = mp.Pool()
>>> # do blocking parallel
>>> y = p.map(doit, x)
>>> y
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]
>>> 
>>> # use an iterator (non-blocking)
>>> y = p.imap(doit, x)
>>> y            
<multiprocess.pool.IMapIterator object at 0x10358d150>
>>> print list(y)
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]
>>> # do asynchronous parallel
>>> y = p.map_async(doit, x)
>>> y
<multiprocess.pool.MapResult object at 0x10358d1d0>
>>> print y.get()
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]
>>>
>>> # or if you like for loops, there's always this…
>>> y = p.imap_unordered(doit, x)
>>> z = []
>>> for i in iter(y):
...   z.append(i)
... 
>>> z
[-1, 0, 3, 8, 15, 24, 35, 48, 63, 80]

最后一种形式是无序迭代器，它往往是最快的...但你不关心结果返回的顺序 - 它们是无序的，而不是保证按照提交的顺序返回。

另请注意，我使用multiprocess（一个分叉）代替multiprocessing ...但纯粹因为multiprocess在处理交互式定义的函数时更好。否则，multiprocessing上面的代码是相同的。

如何并行化这段代码？

1 个答案: