运行多个流程并协调完成

时间:2017-05-24 16:53:44

标签: python parallel-processing popen

我对Popen的工作方式感到有点困惑,我希望这是愚蠢的事情。我永远不会完成,民意调查似乎正在返回奇怪的东西(附加日志)

这是使用提供的实用程序(CSSBACKUP)来备份三重模式(表空间)。

for i in range(len(schematype)):
    schema_base = schemaname + '_' + schematype[i]  # we need this without the trailing space.
    outputstring = base_folder + '\\' + schemaname + '\\' + schema_base + '_' + timestr + '_cssbackup '
    rc = os.unlink(outputstring)   # wont run if there is a backup already
    logstring = base_folder + '\\' + schemaname + '\\' + schema_base + '_' + timestr + '.log'
    exString = "cssbackup " + phys_string + '-schema '+ schema_base + ' ' + '-file ' + outputstring + '-log '+ logstring
    logging.debug(exString)
    processlist.append(subprocess.Popen(exString)) # start a seperate thread for each one, but we don't want to proceed until processlist[].poll == None (thread is complete)
    procdone[i] = False

现在我已经生成了所有进程,我需要将它们同步

while finishit < len(schematype):
    time.sleep(CSTU_CFG.get(hostname).get("logintv"))                                   # need a delay to keep this program from thrashing
    for i in range(len(schematype)):                            # check each of the procs
        if procdone[i] is not True:                                 # if it completed, skip it 
           if processlist[i].poll is not None:                  # if it returns something other than "none" it's still running
                logging.debug('   Running '+ schematype[i] + ' ' + str(processlist[i])+ ' '+ str(time.time() - start_time))
                procdone[i] = False
           else:
                procdone[i] = True                              # None was returned so it's finished
                logging.debug('   Ended '+ schematype[i])           # log it 
                finishit = finishit + 1                         # update the count
                processlist[i].kill                             # kill the process that was running ( Saves memory )

logging.debug('Dump functions complete')        

当我跑这个时,我得不到我所期待的。我期待回报中有一个pid,但我没有看到它。所以我得到的东西对.poll命令没用。

所以即使在它产生的shell消失之后程序也会永远运行。

我遗漏了一些基本的东西。

感谢

11:26:26,133 root,DEBUG Running local 30.014784812927246 11:26:26,133 root,DEBUG Running central 30.014784812927246 11:26:26,133 root,DEBUG Running mngt 30.014784812927246 11:26:56,148 root,DEBUG Running local 60.02956962585449 11:26:56,148 root,DEBUG Running 60.02956962585449 11:26:56,148 root,DEBUG Running mngt 60.02956962585449 11:27:26,162 root,DEBUG运行本地90.04435467720032 11:27:26,162 root,DEBUG Running central 90.04435467720032 11:27:26,162 root,DEBUG Running mngt 90.04435467720032 11:27:56,177 root,DEBUG运行本地120.05913925170898 11:27:56,177 root,DEBUG Running central 120.05913925170898 11:27:56,177 root,DEBUG Running mngt 120.05913925170898 11:28:26,192 root,DEBUG Running local 150.07392406463623

1 个答案:

答案 0 :(得分:2)

你应该调用民意调查。 if processlist[i].poll is not None将始终评估为True,因为processlist[i].poll是函数对象,而不是processlist[i].poll()的结果。

修改 这看起来像是一种非常复杂的方式来做像

这样的事情
p = multiprocessing.Pool(n)
p.map_async(subprocess.call, commands)

作为建议,您可能需要检查多处理模块。