如何在核心之间拆分python工作? (多处理库)

时间:2014-09-22 02:47:44

标签: python multiprocessing

我有一个顺序功能,可以对列表进行排序并执行任务。例如......(这不是实际的代码,但是是偶然的)

def myFunction(list):
   for item in list:
      sublist_a=item[0]
      sublist_b=item[1]
      sublist_c=item[2]
      sublist_d=item[3]
   for row in sublist_a:
      #(do tasks....)
   for row in sublist_b:
      #(do tasks....)
   for row in sublist_c:
      #(do tasks....)
   for row in sublist_d:
      #(do tasks....)
   print "COMPLETE"

所以这个过于简化,但基本上这些列表很大,执行顺序很重要(即。for row in ....),所以我想将它们拆分到我系统上的可用内核之间。 / p>

有人可以建议一种方法吗?

从未使用多处理库,但似乎这可能是最好与python一起使用。

1 个答案:

答案 0 :(得分:2)

您正在寻找multiprocessing.Pool

from multiprocessing import Pool

def function_to_process_a(row):
    return row * 42 # or something similar

# replace 4 by the number of cores that you want to utilize
with Pool(processes=4) as pool:
    # The lists are processed one after another,
    # but the items are processed in parallel.
    processed_sublist_a = pool.map(function_to_process_a, sublist_a)
    processed_sublist_b = pool.map(function_to_process_b, sublist_b)
    processed_sublist_c = pool.map(function_to_process_c, sublist_c)
    processed_sublist_d = pool.map(function_to_process_d, sublist_d)

编辑:正如评论中指出的一样,最好使用这种模式:

from contextlib import closing, cpu_count, Pool

with closing(Pool(processes=cpu_count())) as pool
    pass # do something