Question

在与 multiprocessing.Pipe 一起使用多处理时，我试图创建简单的日志记录。这个想法是让每个正在运行的进程将记录的消息放入队列中，而仅由父进程将这些消息从队列中取出，然后保存到文件中。这样，只有父进程才能访问文件。尽管这在Ubuntu上可以正常运行，但在Windows 10上，保存到文件时会丢失一些日志。将日志打印到控制台可以正常工作，没有日志丢失。我使用 multiprocessing-logging 库。 Anaconda上的Python版本3.6.8。

我正在跟踪问题的根源，并且似乎为每个日志执行了 FileHandler.emit（）。看来问题发生在 multiprocessing 库内部。

import logging
import multiprocessing
from multiprocessing import freeze_support

import attr

import multiprocessing as mp
import multiprocessing_logging

FORMAT = '[%(asctime)s] [%(levelname)s] ' \
         '[%(process)s] [%(process_)s] --- %(message)s'
formatter = logging.Formatter(FORMAT)

fh = logging.FileHandler("test.log")
fh.setFormatter(formatter)
logger = logging.getLogger()
logger.addHandler(fh)
sh = logging.StreamHandler()
sh.setFormatter(formatter)
logger.addHandler(sh)
logger.setLevel(logging.DEBUG)

multiprocessing_logging.install_mp_handler(logger)


@attr.s
class Message:
    id = attr.ib()
    value = attr.ib(repr=False)


def test_func1(name, in_stream, out_stream):
    for a in range(1, 101):
        m = Message(a, a)
        logging.info(m, extra={'process_': name})
        out_stream.send(m)


def test_func2(name, in_stream, out_stream):
    while 1:
        if in_stream.poll(0.1):
            m: Message = in_stream.recv()
            m.value = m.value**2
            logging.warning(m, extra={'process_': name})
            out_stream.send(m)


def test_func3(name, in_stream, out_stream):
    while 1:
        if in_stream.poll(0.1):
            m: Message = in_stream.recv()
            m.value = m.value*2
            logging.error(m, extra={'process_': name})


services = []
@attr.s
class Service:
    name = attr.ib()
    in_stream = attr.ib()
    out_stream = attr.ib()
    func = attr.ib()

    def run(self):
        worker = mp.Process(
            target=self.func,
            args=(
                self.name,
                self.in_stream,
                self.out_stream,
            ),
        )
        worker.start()


if __name__ == '__main__':
    freeze_support()

    prev_reader = None
    next_reader, next_writer = mp.Pipe()
    s1 = Service('f1', prev_reader, next_writer, test_func1)
    services.append(s1)
    prev_reader = next_reader
    next_reader, next_writer = mp.Pipe()
    s2 = Service('f22', prev_reader, next_writer, test_func2)
    services.append(s2)
    prev_reader = next_reader
    next_reader, next_writer = mp.Pipe()
    s3 = Service('f333', prev_reader, next_writer, test_func3)
    services.append(s3)

    for s in services:
        s.run()

部分日志保存到文件中

[2019-08-12 10:09:30,317] [ERROR] [10748] [f333] --- Message(id=31)
[2019-08-12 10:09:30,320] [WARNING] [14060] [f22] --- Message(id=40)
[2019-08-12 10:09:30,311] [INFO] [2828] [f1] --- Message(id=53)
2)
[2019-08-12 10:09:30,320] [WARNING] [14060] [f22] --- Message(id=41)
[2019-08-12 10:09:30,311] [INFO] [2828] [f1] --- Message(id=54)
[2019-08-12 10:09:30,320] [WARNING] [14060] [f22] --- Message(id=42)
[2019-08-12 10:09:30,318] [ERROR] [10748] [f333] --- Message(id=33)

控制台输出中日志的同一部分：

[2019-08-12 10:09:30,317] [ERROR] [10748] [f333] --- Message(id=31)
[2019-08-12 10:09:30,318] [WARNING] [14060] [f22] --- Message(id=33)
[2019-08-12 10:09:30,311] [INFO] [2828] [f1] --- Message(id=52)
[2019-08-12 10:09:30,311] [INFO] [2828] [f1] --- Message(id=53)
[2019-08-12 10:09:30,317] [ERROR] [10748] [f333] --- Message(id=32)
[2019-08-12 10:09:30,318] [WARNING] [14060] [f22] --- Message(id=34)
[2019-08-12 10:09:30,311] [INFO] [2828] [f1] --- Message(id=54)
[2019-08-12 10:09:30,318] [ERROR] [10748] [f333] --- Message(id=33)

请告诉您这里可能是什么问题，以及是否可以在Windows上使用此方法，或者是否应该更改它。

为什么在Windows上将python多处理日志记录保存到文件时会丢失一些日志？

0 个答案: