Question

我正在跟踪documentation进行多进程日志记录，但是我看到在每个子进程中工作程序创建了两个日志。我在某个地方犯了一个愚蠢的错误吗？

环境：

Python 3.6.1 | Anaconda自定义（64位）| （默认值，2017年5月11日，13：25：24）在Win32上的[MSC v.1900 64位（AMD64）]

代码（已编辑，以解决@georgexsh建议的范围问题）：

import logging
import multiprocessing


logger = multiprocessing.log_to_stderr(logging.INFO)


def test(i):
    logger.info(f'worker processing {i}')


if __name__ == '__main__':
    with multiprocessing.Pool() as pool:
        metrics = pool.map(test, range(20))

记录输出：

[INFO/SpawnPoolWorker-2] child process calling self.run()
[INFO/SpawnPoolWorker-2] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 0
[INFO/SpawnPoolWorker-3] child process calling self.run()
[INFO/SpawnPoolWorker-3] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 0
[INFO/SpawnPoolWorker-1] child process calling self.run()
[INFO/SpawnPoolWorker-3] worker processing 1
[INFO/SpawnPoolWorker-2] worker processing 2
[INFO/SpawnPoolWorker-1] child process calling self.run()
[INFO/SpawnPoolWorker-6] child process calling self.run()
[INFO/SpawnPoolWorker-3] worker processing 1
[INFO/SpawnPoolWorker-4] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 2
[INFO/SpawnPoolWorker-5] child process calling self.run()
[INFO/SpawnPoolWorker-7] child process calling self.run()
[INFO/SpawnPoolWorker-1] worker processing 3
[INFO/SpawnPoolWorker-6] child process calling self.run()
[INFO/SpawnPoolWorker-3] worker processing 4
[INFO/SpawnPoolWorker-4] child process calling self.run()
[INFO/SpawnPoolWorker-2] worker processing 5
[INFO/SpawnPoolWorker-5] child process calling self.run()
[INFO/SpawnPoolWorker-7] child process calling self.run()
[INFO/SpawnPoolWorker-1] worker processing 3
[INFO/SpawnPoolWorker-6] worker processing 6
[INFO/SpawnPoolWorker-3] worker processing 4
...
[INFO/SpawnPoolWorker-5] worker processing 16
[INFO/SpawnPoolWorker-2] worker processing 12
[INFO/SpawnPoolWorker-7] worker processing 17
[INFO/SpawnPoolWorker-1] worker processing 18
[INFO/SpawnPoolWorker-6] worker processing 13
[INFO/SpawnPoolWorker-3] worker processing 19
[INFO/SpawnPoolWorker-8] worker processing 14
[INFO/SpawnPoolWorker-4] worker processing 15
[INFO/SpawnPoolWorker-5] worker processing 16
[INFO/SpawnPoolWorker-7] worker processing 17
[INFO/SpawnPoolWorker-1] worker processing 18
[INFO/SpawnPoolWorker-3] worker processing 19
[INFO/SpawnPoolWorker-2] process shutting down
[INFO/SpawnPoolWorker-6] process shutting down
[INFO/MainProcess] process shutting down

Answer 1

将logger = multiprocessing.log_to_stderr()移到全局范围，而不是在工作函数内部。确保只调用一次。因为每次log_to_stderr被调用，it will add a new handler to the logger：

def test(i):
    logger.info('worker processing %s', i)

if __name__ == '__main__':
    logger = multiprocessing.log_to_stderr(logging.INFO)

请注意，在Windows下，由于创建子进程以重建上下文时没有fork()，the whole module get executed again，因此您可以使用Pool's initializer初始化记录器，该记录器仅在子进程：

logger = None

def test(i):
    logger.info('worker processing %s', i)

def initializer(level):
    global logger
    logger = multiprocessing.log_to_stderr(level)

if __name__ == '__main__':
    pool = multiprocessing.Pool(4, initializer=initializer, initargs=(logging.INFO,))
    metrics = pool.map(test, range(20))

Windows上重复的Python多处理log_to_stderr

1 个答案: