Question

在从luigi运行应用程序时，我将日志记录定向到stdout和文件。这适用于workers = 1。但是，只要我设置workers = 4，我就没有应用程序记录。

多个工人可以记录吗？

Answer 1

似乎在路易吉没有简单的解决方案，但以下工作。唯一的事情是你必须记住在每个run方法的开头调用“enable”。我还没有找到避免这种情况的解决方案。

import logging
import luigi
from luigi.interface import build, setup_interface_logging

from multiprocessing import Queue
from logging.handlers import QueueHandler, QueueListener

########### enable multiprocess logging ############################

q = Queue()

def run(tasks, *args, **kwargs):
    """ run tasks with queuelistener to handle logging """
    setup_interface_logging.has_run = True
    workers = kwargs.get("workers", 1) > 1
    if workers:
        log=logging.getLogger()
        listener = QueueListener(q, *log.handlers)
        listener.start()
        build(tasks, *args, **kwargs)
        listener.stop()
    else:
        build(tasks, *args, **kwargs)

class Task(luigi.Task):
    """ redirect logging to queue """
    def __init__(self):
        """ add q to process """
        super().__init__()
        self.q = q

    def enable(self):
        """ call at start of each run process to initialise settings and redirect logging to queue """

        # for 1 worker leave settings alone
        log = logging.getLogger()
        if log.handlers:
            return

        # for multiple workers load settings but replace handlers with queue
        from logcon import log
        log.handlers = []
        log.addHandler(QueueHandler(self.q))

######################################################################

class Test(Task):

    def complete(self):
        return False

    def run(self):
        self.enable()

        log = logging.getLogger()
        log.debug("running")
        log.info("running")
        log.warning("running")

        log = logging.getLogger("runlog")
        log.debug("running")
        log.info("running")
        log.warning("running")

        log = logging.getLogger("luigi-interface")
        log.debug("running")
        log.info("running")
        log.warning("running")

    def requires(self):
        return []

如何在多个工作人员的luigi中配置日志记录？

1 个答案: