Question

if __name__ == '__main__':

    MATCH_ID = str(doc_ref2.id)

    MATCH_ID_TEAM = doc_ref3.id

    with concurrent.futures.ProcessPoolExecutor(max_workers=30) as executor:
        results = list(executor.map(ESPNPlayerFree, teamList1))

    MATCH_ID_TEAM = str(doc_ref4.id)

    with concurrent.futures.ProcessPoolExecutor(max_workers=30) as executor:
        results = list(executor.map(ESPNPlayerFree, teamList2))

当我打印MATCH_ID_TEAM时，它会打印值。但是在此过程中，它显示了默认值，我在顶部将其设置为空。

如何更新所有进程的变量值？

ESPNPlayerFree is a class that takes `id` as an argument. So `teamList1` and `teamList2` are list of ids to initialize my objects.

MATCH_ID和MATCH_ID_TEAM是我的课程ESPNPlayerFree中使用的变量

操作系统 Windows 10 64位

IDE Pycharm

Python版本 3.6.1

Answer 1

几天前，我正在整理@furas在他的评论中的位置。实际上，最简单的方法是只传递您班上需要的所有内容以及.map()。 executor.map()期望迭代器，对于在您的工作程序中进行的每个函数调用，这些迭代器将压缩为一个参数元组。

您显然需要MATCH_ID和MATCH_ID_TEAM才能在整个工作中保持相同，即一个呼叫executor.map()。您面临的挑战是两个都是可迭代项（字符串），但是您需要将它们全部复制为，并且经常要与团队列表可迭代项中的每个项目匹配。

因此，您将字符串与团队ID列表一起传递给.map()时，只需用itertools.repeat()包装这些字符串。默认情况下，itertools.repeat()返回所传递对象的无限迭代器。然后ProcessPoolExecutor在内部使用zip()组合所有可迭代项中的项作为参数。

import concurrent.futures
import multiprocessing
from itertools import repeat


class ESPNPlayerFree:
    def __init__(self, team_id, match_id, match_id_team):
        self.teams_id = team_id
        self.match_id = match_id
        self.match_id_team = match_id_team
        print(
            multiprocessing.current_process().name,
            self.teams_id, self.match_id, self.match_id_team
        )


if __name__ == '__main__':

    teams1 = [f"id{i}" for i in range (10)]
    teams2 = [f"id{i}" for i in range(10, 20)]

    with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:

        MATCH_ID = 'doc_ref2.id'
        MATCH_ID_TEAM = 'doc_ref3.id'

        results = list(
            executor.map(
                ESPNPlayerFree,
                teams1,
                repeat(MATCH_ID),
                repeat(MATCH_ID_TEAM),
            )
        )

        print("--- new MATCH_ID_TEAM ---")
        MATCH_ID_TEAM = 'doc_ref4.id'

        results = list(
            executor.map(
                ESPNPlayerFree,
                teams2,
                repeat(MATCH_ID),
                repeat(MATCH_ID_TEAM),
            )
        )

输出：

ForkProcess-1 id0 doc_ref2.id doc_ref3.id
ForkProcess-2 id1 doc_ref2.id doc_ref3.id
ForkProcess-3 id2 doc_ref2.id doc_ref3.id
ForkProcess-4 id3 doc_ref2.id doc_ref3.id
ForkProcess-1 id4 doc_ref2.id doc_ref3.id
ForkProcess-3 id5 doc_ref2.id doc_ref3.id
ForkProcess-2 id6 doc_ref2.id doc_ref3.id
ForkProcess-4 id7 doc_ref2.id doc_ref3.id
ForkProcess-3 id8 doc_ref2.id doc_ref3.id
ForkProcess-1 id9 doc_ref2.id doc_ref3.id
--- new MATCH_ID_TEAM ---
ForkProcess-1 id10 doc_ref2.id doc_ref4.id
ForkProcess-3 id11 doc_ref2.id doc_ref4.id
ForkProcess-2 id12 doc_ref2.id doc_ref4.id
ForkProcess-4 id13 doc_ref2.id doc_ref4.id
ForkProcess-1 id14 doc_ref2.id doc_ref4.id
ForkProcess-3 id15 doc_ref2.id doc_ref4.id
ForkProcess-2 id16 doc_ref2.id doc_ref4.id
ForkProcess-4 id17 doc_ref2.id doc_ref4.id
ForkProcess-2 id18 doc_ref2.id doc_ref4.id
ForkProcess-1 id19 doc_ref2.id doc_ref4.id

Process finished with exit code 0

对于第二项工作，使用新的MATCH_ID_TEAM，您不必再次重新创建ProcessPoolExecutor，只要您需要，只要停留在上下文管理器中，就可以再次使用现有的内容

使用ProcessPoolExecutor时更新变量

1 个答案: