Question

我的应用程序使用multiprocessing.pool来并行化计算。现在我想添加日志记录功能。代码（不幸的是）需要在Windows上运行。我在stackoverflow找到了一个相关帖子，但它没有用。我认为包 multiprocessing_logging 不支持池。

这是我的代码：

from multiprocessing_logging import install_mp_handler

def main(): # main function
    filename = "XXX" + datetime.datetime.now().strftime('%Y-%m-%d-%H.%M.%S') + ".log"

    log_file = os.path.abspath(os.path.join('logs',filename))
    multiprocessing.freeze_support() # support multiprocessing

    logging.basicConfig(filename=log_file,
                        filemode='a',
                        format='%(asctime)s:%(msecs)d (%(processName)s) %(levelname)s %(name)s \t %(message)s',
                        datefmt='%H:%M:%S',
                        level=logging.DEBUG)

    logger.info("Start application")

def run(): # main exection
    logger.info("Generate outputs for every metrics")
    num_cores = multiprocessing.cpu_count()
    logger.info("Output Generation execute on " + str(num_cores) + " cores" )

    pool = Pool(num_cores, initializer=install_mp_handler )
    processed_metrics = pool.map(_generate_outputs, metrics_list)
    pool.close()
    pool.join()
    map(_create_report,processed_metrics)

辅助函数 _generate_outputs 和 _create_report 的实现与问题无关。当我执行代码时，模块从主进程生成的日志被正确存储，但不是来自子进程。

[编辑]
我根据评论改变了我的代码。现在，我的代码如下所示：

    num_cores = multiprocessing.cpu_count()
    logger.info("Output Generation execute on " + str(num_cores) + " cores" )
    install_mp_handler()
    pool = Pool(num_cores, initializer=install_mp_handler )
    processed_metrics = pool.map(_generate_outputs, metrics_list)
    pool.close()
    pool.join()
    map(_create_report,processed_metrics)

但是，仍未捕获来自子进程的日志。程序终止后，我看到一个错误：

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\multiprocessing_logging.py", line 64, in _receive
    record = self.queue.get(timeout=0.2)
  File "C:\Python27\lib\multiprocessing\queues.py", line 131, in get
    if not self._poll(timeout):
IOError: [Errno 109] The pipe has been ended
Exception in thread mp-handler-0:
Traceback (most recent call last):
  File "C:\Python27\lib\threading.py", line 801, in __bootstrap_inner
    self.run()
  File "C:\Python27\lib\threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:\Python27\lib\site-packages\multiprocessing_logging.py", line 62, in _receive
    while not (self._is_closed and self.queue.empty()):
  File "C:\Python27\lib\multiprocessing\queues.py", line 146, in empty
    return not self._poll()
IOError: [Errno 109] The pipe has been ended

关键要求是程序需要在Windows上运行。

Answer 1

您需要在install_mp_handler()实例化之前调用Pool()。

...
install_mp_handler()
pool = Pool(num_cores, initializer=install_mp_handler)
...

最后，所有这些都归结为通过队列传输到集中式日志处理程序的日志记录，请查看https://hastebin.com/baguteruki.cs，它清楚地了解了该技术。

python多处理池和日志记录

1 个答案: