我有一个顺序功能,可以对列表进行排序并执行任务。例如......(这不是实际的代码,但是是偶然的)
def myFunction(list):
for item in list:
sublist_a=item[0]
sublist_b=item[1]
sublist_c=item[2]
sublist_d=item[3]
for row in sublist_a:
#(do tasks....)
for row in sublist_b:
#(do tasks....)
for row in sublist_c:
#(do tasks....)
for row in sublist_d:
#(do tasks....)
print "COMPLETE"
所以这个过于简化,但基本上这些列表很大,执行顺序很重要(即。for row in ....
),所以我想将它们拆分到我系统上的可用内核之间。 / p>
有人可以建议一种方法吗?
从未使用多处理库,但似乎这可能是最好与python一起使用。
答案 0 :(得分:2)
您正在寻找multiprocessing.Pool
from multiprocessing import Pool
def function_to_process_a(row):
return row * 42 # or something similar
# replace 4 by the number of cores that you want to utilize
with Pool(processes=4) as pool:
# The lists are processed one after another,
# but the items are processed in parallel.
processed_sublist_a = pool.map(function_to_process_a, sublist_a)
processed_sublist_b = pool.map(function_to_process_b, sublist_b)
processed_sublist_c = pool.map(function_to_process_c, sublist_c)
processed_sublist_d = pool.map(function_to_process_d, sublist_d)
编辑:正如评论中指出的一样,最好使用这种模式:
from contextlib import closing, cpu_count, Pool
with closing(Pool(processes=cpu_count())) as pool
pass # do something