多处理子过程中的混乱

时间:2019-03-26 18:00:57

标签: python multiprocessing subprocess

使用主持人脚本(请参见下文)以线性方式很好地运行了一些计算后,我努力了 在尝试进行多处理时使其执行。似乎每个CPU内核都在此列表集(testRegister)中运行并启动计算,即使另一个内核早先(在同一会话中)已经执行了此任务。如何防止这种混乱的行为?这是我第一次尝试通过Python调用多个处理器。

更正:最初的帖子没有显示测试是一个字符串,该字符串包含调用参数为m1和m2的“内部脚本”,而固定参数arg1和arg2仅属于此“内部脚本”。

#!/usr/bin/env python3
import os
import subprocess as sub
import sys
import multiprocessing
fileRegister = []
testRegister = []


def fileCollector():
    for file in os.listdir("."):
        if file.endswith(".xyz"):
            fileRegister.append(file)
    fileRegister.sort()
    return fileRegister


def testSetup():
    data = fileRegister
    while len(data) > 1:
        for entry in fileRegister[1:]:
            m0 = str(fileRegister[0])
            m1 = str(entry)
            test =  str("python foo.py ") + str(m1) + str(" ") + str(m2) +\
                    str(" --arg1 --arg2")  # formulate test condition
            testRegister.append(test)
            testRegister.sort()
        del data[0]
    return testRegister


def shortAnalysator():
    for entry in testRegister:
        print(str(entry))
        sub.call(entry, shell=True)
        del testRegister[0]


def polyAnalysator():
    # apparently each CPU core works as if the register were not shared
    # reference: https://docs.python.org/3.7/library/multiprocessing.html
    if __name__ == '__main__':
        jobs = []
        for i in range(3):   # safety marging to not consume all CPU
            p = multiprocessing.Process(target=shortAnalysator)
            jobs.append(p)
            p.start()


fileCollector()
testSetup()
shortAnalysator()       # proceeding expectably on one CPU (slow)
# polyAnalysator()        # causing irritation
sys.exit()```

1 个答案:

答案 0 :(得分:1)

您的polyAnalysator正在运行shortAnalysator三次。尝试如下更改polyAnalysator,然后添加f方法。这使用multiprocessing Pool

from multiprocessing import Pool

def f(test):
    sub.call(test, shell=True)


def polyAnalysator():
    # apparently each CPU core works as if the register were not shared
    # reference: https://docs.python.org/3.7/library/multiprocessing.html
    with Pool(3) as p:
        p.map(f, testRegister)