使用主持人脚本(请参见下文)以线性方式很好地运行了一些计算后,我努力了 在尝试进行多处理时使其执行。似乎每个CPU内核都在此列表集(testRegister)中运行并启动计算,即使另一个内核早先(在同一会话中)已经执行了此任务。如何防止这种混乱的行为?这是我第一次尝试通过Python调用多个处理器。
更正:最初的帖子没有显示测试是一个字符串,该字符串包含调用参数为m1和m2的“内部脚本”,而固定参数arg1和arg2仅属于此“内部脚本”。
#!/usr/bin/env python3
import os
import subprocess as sub
import sys
import multiprocessing
fileRegister = []
testRegister = []
def fileCollector():
for file in os.listdir("."):
if file.endswith(".xyz"):
fileRegister.append(file)
fileRegister.sort()
return fileRegister
def testSetup():
data = fileRegister
while len(data) > 1:
for entry in fileRegister[1:]:
m0 = str(fileRegister[0])
m1 = str(entry)
test = str("python foo.py ") + str(m1) + str(" ") + str(m2) +\
str(" --arg1 --arg2") # formulate test condition
testRegister.append(test)
testRegister.sort()
del data[0]
return testRegister
def shortAnalysator():
for entry in testRegister:
print(str(entry))
sub.call(entry, shell=True)
del testRegister[0]
def polyAnalysator():
# apparently each CPU core works as if the register were not shared
# reference: https://docs.python.org/3.7/library/multiprocessing.html
if __name__ == '__main__':
jobs = []
for i in range(3): # safety marging to not consume all CPU
p = multiprocessing.Process(target=shortAnalysator)
jobs.append(p)
p.start()
fileCollector()
testSetup()
shortAnalysator() # proceeding expectably on one CPU (slow)
# polyAnalysator() # causing irritation
sys.exit()```
答案 0 :(得分:1)
您的polyAnalysator
正在运行shortAnalysator
三次。尝试如下更改polyAnalysator
,然后添加f
方法。这使用multiprocessing Pool
:
from multiprocessing import Pool
def f(test):
sub.call(test, shell=True)
def polyAnalysator():
# apparently each CPU core works as if the register were not shared
# reference: https://docs.python.org/3.7/library/multiprocessing.html
with Pool(3) as p:
p.map(f, testRegister)