我正在试图找出如何通过使用多处理来加速网格搜索在Python中获取2输入（较低阈值和较高阈值）的函数。我检查了http://pymotw.com/2/multiprocessing/basics.html并尝试使用以下内容：
import multiprocessing
lower_threshold = range(0,100)
upper_threshold = range(100,200)
def worker():
bestAccuracy = 0
bestLowerThreshold = 0
bestUpperThreshold = 0
for lowerThreshold in lower_threshold:
for upperThreshold in upper_threshold:
accuracy = someOtherFunction(lower_threshold,upper_threshold)
if accuracy > bestAccuracy:
bestAccuracy = accuracy
bestLowerTheshold = lowerThreshold
bestUpperThreshold = upperThreshold
return bestAccuracy, bestLowerThreshold, bestUpperThreshold
if __name__ == '__main__':
jobs = []
for i in range(8):
p = multiprocessing.Process(target=worker)
jobs.append(p)
p.start()
当我运行此代码时，我最终得到的8个核心进程为我的函数运行相同的参数/输入。有什么方法可以指定核心进程应该使用不同的参数吗？提前，谢谢你的帮助。
答案 0 :(得分：2)
您可以使用itertools.product创建包含嵌套for循环中所有(lowerThreshHold, upperThreshold)对的2元素元组的迭代器，然后将迭代器拆分为8个块，每个块传递一个池中的进程。然后，您只需对每个工作人员获得的结果进行排序，以获得最佳的整体结果。我还建议使用multiprocessing.Pool代替multiprocessing.Process，以简化操作。
import multiprocessing
import itertools
def get_chunks(iterable, chunks=1):
# This is from http://stackoverflow.com/a/2136090/2073595
lst = list(iterable)
return [lst[i::chunks] for i in xrange(chunks)]
def someOtherFunction(lower, upper):
return (lower + upper) / 2
def worker(pairs):
best_accuracy = best_lower = best_upper = 0
for lower_threshold, upper_threshold in pairs:
accuracy = someOtherFunction(lower_threshold, upper_threshold)
if accuracy > best_accuracy:
best_accuracy = accuracy
best_lower = lower_threshold
best_upper = upper_threshold
return best_accuracy, best_lower, best_upper
if __name__ == '__main__':
jobs = []
pairs = itertools.product(xrange(0, 100), xrange(100, 200))
chunked_pairs = get_chunks(pairs, chunks=multiprocessing.cpu_count())
pool = multiprocessing.Pool()
results = pool.map(worker, chunked_pairs)
pool.close()
pool.join()
print results
# Now combine the results
sorted_results = reversed(sorted(results, key=lambda x: x[0]))
print next(sorted_results) # Winner
输出：
[(147, 98, 196), (148, 99, 197), (148, 98, 198), (149, 99, 199)]
(149, 99, 199)
答案 1 :(得分：0)
通常，如果您拥有多个工作人员池和一长串输入，则多处理工作效果最佳。每批输入并行处理，结果返回给调用者。然后池运行下一批，依此类推，直到消耗完所有输入。
以下代码在lower_threshold范围内传递，每个代码都重新使用upper_threshold：
import multiprocessing, random
upper_threshold = range(10,20)
def someOtherFunction(a,b):
print 'ding:',a,b
return random.randint(10, 100)
def worker(thresh):
bestAccuracy = 0
bestLowerThreshold = 0
bestUpperThreshold = 0
for upperThreshold in upper_threshold:
accuracy = someOtherFunction(thresh, upper_threshold)
if accuracy > bestAccuracy:
bestAccuracy = accuracy
bestLowerTheshold = thresh
bestUpperThreshold = upperThreshold
return bestAccuracy, bestLowerThreshold, bestUpperThreshold
if __name__ == '__main__':
lower_threshold = range(0,10)
pool = multiprocessing.Pool()
print pool.map( worker, [[th] for th in lower_threshold] )
pool.close()
pool.join()
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [0] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [4] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [6] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [1] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [5] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [7] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [3] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [9] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [2] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
ding: [8] [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[(80, 0, 11), (98, 0, 17), (96, 0, 17), (89, 0, 17), (96, 0, 17), (95, 0, 10), (95, 0, 18), (100, 0, 18), (90, 0, 18), (97, 0, 17)]