Question

我想使用多处理运行以下功能：

def bruteForcePaths3(paths, availableNodes):

    results = []

    #start by taking each combination 2 at a time, then 3, etc
    for i in range(1,len(availableNodes)+1):
        print "combo number: %d" % i

        currentCombos = combinations(availableNodes, i)

        for combo in currentCombos:
            #get a fresh copy of paths for this combiniation
            currentPaths = list(paths)
            currentRemainingPaths = []
            # print combo

            for node in combo:
                #determine better way to remove nodes, for now- if it's in, we remove
                currentRemainingPaths = [path for path in currentPaths if not (node in path)]
                currentPaths = currentRemainingPaths

            #if there are no paths left
            if len(currentRemainingPaths) == 0:
                #save this combination
                print combo
                results.append(frozenset(combo))

    return results

基于其他一些帖子（Combining itertools and multiprocessing?），我尝试通过以下方式对此进行多处理：

def grouper_nofill(n, iterable):
        it=iter(iterable)
        def take():
            while 1: yield list(islice(it,n))
        return iter(take().next,[])

    def mp_bruteForcePaths(paths, availableNodes):

        pool = multiprocessing.Pool(4)
        chunksize=256
        async_results=[]


        def worker(paths,combos, out_q):
            """ The worker function, invoked in a process. 'nums' is a
                list of numbers to factor. The results are placed in
                a dictionary that's pushed to a queue.
            """
            results = bruteForcePaths2(paths, combos)
            print results
            out_q.put(results)


        for i in range(1,len(availableNodes)+1):
            currentCombos = combinations(availableNodes, i)
            for finput in grouper_nofill(chunksize,currentCombos):

                args = (paths, finput)
                async_results.extend(pool.map_async(bruteForcePaths2, args).get())

                print async_results

    def bruteForcePaths2(args):
        paths, combos = args
        results = []

        for combo in combos:
            #get a fresh copy of paths for this combiniation
            currentPaths = list(paths)
            currentRemainingPaths = []
            # print combo

            for node in combo:
                #determine better way to remove nodes, for now- if it's in, we remove
                currentRemainingPaths = [path for path in currentPaths if not (combo in path)]
                currentPaths = currentRemainingPaths

            #if there are no paths left
            if len(currentRemainingPaths) == 0:
                #save this combination
                print combo
                results.append(frozenset(combo))

        return results

我需要能够将2个参数传递给bruteforce函数。我收到错误： “打开包装的价值太多了”

所以3部分问题：如何通过nproc cpu拆分组合迭代器来对bruteforce函数进行多处理？我怎样才能传递两个参数 - 路径和组合？我如何得到结果（想想mpa_async应该为我做这个）？

感谢。

Answer 1

此

args = (paths, finput)
pool.map_async(bruteForcePaths2, args)

进行这两次调用，这不是你的意图：

bruteForcePaths2(paths)
bruteForcePaths2(finput)

您可以使用apply_async代替向池中提交单个函数调用。另请注意，如果您立即调用get，它将等待结果，并且您无法从多处理中获得任何好处。

你可以这样做：

for i in range(1,len(availableNodes)+1):
    currentCombos = combinations(availableNodes, i)
    for finput in grouper_nofill(chunksize,currentCombos):

        args = (paths, finput)
        async_results.append(pool.apply_async(bruteForcePaths2, args))

results = [x.get() for x in async_results]

具有两个参数的多进程itertool组合

1 个答案: