Question

我有一个可执行文件，我需要经常运行，使用不同的参数。为此，我使用多处理模块编写了一个小的Python（2.7）包装器，遵循给定here的模式。

我的代码如下所示：

try:
     logging.info("starting pool runs")
     pool.map(run_nlin, params)
     pool.close()
 except KeyboardInterrupt:
     logging.info("^C pressed")
     pool.terminate()
 except Exception, e:
     logging.info("exception caught: ", e)
     pool.terminate()
 finally:
     time.sleep(5)
     pool.join()
     logging.info("done")

我的工作人员在这里：

class KeyboardInterruptError(Exception): pass

def run_nlin((path_config, path_log, path_nlin, update_method)):
    try:
        with open(path_log, "w") as log_:
            cmdline = [path_nlin, path_config]
            if update_method:
                cmdline += [update_method, ]
            sp.call(cmdline, stdout=log_, stderr=log_)
    except KeyboardInterrupt:
        time.sleep(5)
        raise KeyboardInterruptError()
    except:
        raise

path_config是二进制程序的配置文件的路径;在那里，例如运行程序的日期。

当我启动包装器时，一切看起来都很好。但是，当我按^C时，包装器脚本似乎在终止之前从池中启动了另外的numproc个进程。例如，当我在1-10天启动脚本时，我可以在ps aux输出中看到二进制程序的两个实例正在运行（通常是第1天和第3天）。现在，当我按^C时，包装器脚本退出，第1天和第3天的二进制程序消失了，但是有新的二进制程序在第5天和第7天运行。

所以对我来说，似乎Pool在最终死亡之前启动了另一个numproc进程。

任何想法在这里发生了什么，我能做些什么呢？

Answer 1

在this page上，多处理模块的作者Jesse Noller表明，处理KeyboardInterrupt的正确方法是让子进程返回 - 而不是重新加载异常。这允许主进程终止池。

但是，如下面的代码所示，主要流程在{em>之后 except KeyboardInterrupt生成pool.map生成的所有任务之后才会到达run_nlin块。这就是为什么（我相信）在按下Ctrl-C后，您会看到对工作人员函数multiprocessing.Event的额外调用。

一种可能的解决方法是，如果已设置import logging import multiprocessing as mp import time logger = mp.log_to_stderr(logging.WARNING) def worker(x): try: if not terminating.is_set(): logger.warn("Running worker({x!r})".format(x = x)) time.sleep(3) else: logger.warn("got the message... we're terminating!") except KeyboardInterrupt: logger.warn("terminating is set") terminating.set() return x def initializer(terminating_): # This places terminating in the global namespace of the worker subprocesses. # This allows the worker function to access `terminating` even though it is # not passed as an argument to the function. global terminating terminating = terminating_ def main(): terminating = mp.Event() result = [] pool = mp.Pool(initializer=initializer, initargs=(terminating, )) params = range(12) try: logger.warn("starting pool runs") result = pool.map(worker, params) pool.close() except KeyboardInterrupt: logger.warn("^C pressed") pool.terminate() finally: pool.join() logger.warn('done: {r}'.format(r = result)) if __name__ == '__main__': main()，则测试所有工作人员功能。如果事件已经设定，那么让工人提前纾困，否则，继续进行长时间的计算。

% test.py
[WARNING/MainProcess] starting pool runs
[WARNING/PoolWorker-1] Running worker(0)
[WARNING/PoolWorker-2] Running worker(1)
[WARNING/PoolWorker-3] Running worker(2)
[WARNING/PoolWorker-4] Running worker(3)

运行脚本会产生：

terminating

按下Ctrl-C;每个工作人员都设置C-c C-c[WARNING/PoolWorker-4] terminating is set [WARNING/PoolWorker-2] terminating is set [WARNING/PoolWorker-3] terminating is set [WARNING/PoolWorker-1] terminating is set事件。我们真的只需要设置一个，但是尽管效率很低，但这仍然有效。

pool.map

现在，[WARNING/PoolWorker-4] got the message... we're terminating! [WARNING/PoolWorker-2] got the message... we're terminating! [WARNING/PoolWorker-1] got the message... we're terminating! [WARNING/PoolWorker-2] got the message... we're terminating! [WARNING/PoolWorker-4] got the message... we're terminating! [WARNING/PoolWorker-2] got the message... we're terminating! [WARNING/PoolWorker-1] got the message... we're terminating! [WARNING/PoolWorker-3] got the message... we're terminating!排队的所有其他任务都会运行：

except KeyboardInterrupt

最后，主进程到达[WARNING/MainProcess] ^C pressed [WARNING/MainProcess] done: []块。

{{1}}

multiprocessing.Pool在Linux / Python2.7上的terminate（）之后产生新的childern？

1 个答案: