Question

使用主持人脚本（请参见下文）以线性方式很好地运行了一些计算后，我努力了在尝试进行多处理时使其执行。似乎每个CPU内核都在此列表集（testRegister）中运行并启动计算，即使另一个内核早先（在同一会话中）已经执行了此任务。如何防止这种混乱的行为？这是我第一次尝试通过Python调用多个处理器。

更正：最初的帖子没有显示测试是一个字符串，该字符串包含调用参数为m1和m2的“内部脚本”，而固定参数arg1和arg2仅属于此“内部脚本”。

#!/usr/bin/env python3
import os
import subprocess as sub
import sys
import multiprocessing
fileRegister = []
testRegister = []


def fileCollector():
    for file in os.listdir("."):
        if file.endswith(".xyz"):
            fileRegister.append(file)
    fileRegister.sort()
    return fileRegister


def testSetup():
    data = fileRegister
    while len(data) > 1:
        for entry in fileRegister[1:]:
            m0 = str(fileRegister[0])
            m1 = str(entry)
            test =  str("python foo.py ") + str(m1) + str(" ") + str(m2) +\
                    str(" --arg1 --arg2")  # formulate test condition
            testRegister.append(test)
            testRegister.sort()
        del data[0]
    return testRegister


def shortAnalysator():
    for entry in testRegister:
        print(str(entry))
        sub.call(entry, shell=True)
        del testRegister[0]


def polyAnalysator():
    # apparently each CPU core works as if the register were not shared
    # reference: https://docs.python.org/3.7/library/multiprocessing.html
    if __name__ == '__main__':
        jobs = []
        for i in range(3):   # safety marging to not consume all CPU
            p = multiprocessing.Process(target=shortAnalysator)
            jobs.append(p)
            p.start()


fileCollector()
testSetup()
shortAnalysator()       # proceeding expectably on one CPU (slow)
# polyAnalysator()        # causing irritation
sys.exit()```

Answer 1

您的polyAnalysator正在运行shortAnalysator三次。尝试如下更改polyAnalysator，然后添加f方法。这使用multiprocessing Pool：

from multiprocessing import Pool

def f(test):
    sub.call(test, shell=True)


def polyAnalysator():
    # apparently each CPU core works as if the register were not shared
    # reference: https://docs.python.org/3.7/library/multiprocessing.html
    with Pool(3) as p:
        p.map(f, testRegister)

多处理子过程中的混乱

1 个答案: