在我的程序中，我想启动多个进程并将每个进程的日志消息收集到一个集中式消息传递队列中。 我创建了一个专用的类'LogMaster'，其作用是从进程中收集日志并通过流处理程序和文件处理程序将其打印出来。 然后，我尝试启动进程，但是由于我的日志没有打印/收集日志消息，所以处理不当。
这是下面的代码，以及启动脚本时的输出结果。 我试图通过打印和写入日志文件插入调试消息来确定问题出在哪里。 “ 1-start_process：打印项正在运行！”但是“ 2-process_func：打印字词有效！”从来没有显示，我不明白为什么。 我恳求您的帮助：）
输出
[user@mydev dev]$ python main.py
2019-03-06 13:05:12,483 INFO MyGraph started
1 - start_process: Print term is working!
2019-03-06 13:05:12,483 INFO 1 - start_process: LogMaster.info is working too!
2019-03-06 13:05:12,483 INFO Start process 'graph.GraphGenerator'
2019-03-06 13:05:12,483 INFO MyGraph ended
main.py
#!/usr/bin/python2
from logger import set_logging, LogMaster
from graph import GraphGenerator
SERVICE_NAME = 'MyGraph'
set_logging(SERVICE_NAME)
LogMaster.start_logging()
LogMaster.info("{} started".format(SERVICE_NAME ))
service = GraphGenerator()
service.start_process()
LogMaster.info("{} ended".format(SERVICE_NAME))
LogMaster.stop_logging()
graph.py 实现了GraphGenerator（）类，该类将负责启动多个进程
from multiprocessing import Process, Queue, Value, Lock
from logger import LogMaster, set_logging
import traceback
class GraphGenerator():
def __init__(self):
pass
def start_process(self):
LogMaster.print_term("1 - start_process: Print term is working!")
LogMaster.info("1 - start_process: LogMaster.info is working too!")
p = Process(target=self.process_func, args=(LogMaster.logging_queue,))
LogMaster.info("Start process '{}'".format(self.__class__))
p.start()
def process_func(self, logging_queue):
LogMaster.print_term("2 - process_func: Print term is working!")
LogMaster.info("2 - process_func: LogMaster.info is working too!")
try:
LogMaster.set_logging_queue(logging_queue)
LogMaster.print_term("process_func: Print term is working!")
LogMaster.info("process_func: LogMaster.info is working too!")
except Exception as e:
print("Ex=================")
LogMaster.info(traceback.format_exc())
logger.py 实现负责记录的LogMaster（）类
from __future__ import print_function
import os
import sys
import logging
from multiprocessing import Queue
from threading import Thread
class LogMaster(object):
logging_queue = None
logging_thread = None
@classmethod
def start_logging(self):
# create pipe to centralise messages
self.logging_queue = Queue()
self.logging_thread = Thread(target=self.logging_func)
self.logging_thread.daemon = True
self.logging_thread.start()
@classmethod
def set_logging_queue(self, q):
self.logging_queue = q
@classmethod
def stop_logging(self):
self.logging_queue.put(None)
self.logging_thread.join()
@classmethod
def print_term(self, msg, end='\n'):
self.logging_queue.put(("print", msg, end))
@classmethod
def log(self, loglevel, msg):
#print(loglevel, msg)
self.logging_queue.put(("log", loglevel, msg))
@classmethod
def info(self, msg):
self.log(logging.INFO, msg)
@classmethod
def logging_func(self):
while True:
item = self.logging_queue.get()
if item == None:
break
elif item[0] == "print":
print(item[1], end=item[2])
sys.stdout.flush()
pass
elif item[0] == "log":
logging.log(item[1], item[2])
def set_logging(logfile_name, verbose=False):
# == General log ==
logger = logging.getLogger()
logger.setLevel(logging.INFO)
# File handler
fh = logging.FileHandler(logfile_name)
fh.setLevel(logging.INFO)
# Stream handler
sh = logging.StreamHandler()
sh.setLevel(logging.INFO)
sh.createLock()
#Formatter
formatter = logging.Formatter('%(asctime)s\t%(levelname)s\t%(message)s')
fh.setFormatter(formatter)
sh.setFormatter(formatter)
# add the handlers to the logger
logger.addHandler(fh)
logger.addHandler(sh)
答案 0 :(得分：0)
您的示例存在计时问题。 service.start_process()几乎立即返回； LogMaster.info("{} ended".format(SERVICE_NAME))也是如此。 LogMaster.stop_logging()在子进程的设置完成之前被调用。您会在孩子开始执行之前将logging_thread拆掉。它仍然可以执行，但是到那时，队列上没有任何监听了。消息迷路了。
您必须提出一个计划，在何时何地同步您的流程。快速解决方案是在p.join()中的p.start()之后简单地添加GraphGenerator.start_process。不过，不确定是否要让主进程阻塞该方法的执行。
下面的代码示例中显示了另一个选项。
在符合POSIX的操作系统上，无需将队列传递到process_func。在这些操作系统上，通过基本复制父级来创建子级，子级进程中的LogMaster类对象是主进程中父级对象的副本，它已经具有队列。
不过，您可以稍微重构LogMaster。该类本身可以是threading.Thread的子类，并且set_logging中的指令可以添加到LogMaster中并作为初始化的一部分执行。
这样，实例化类可以设置日志记录并启动队列的侦听器，而访问类本身的静态/类方法将仅用于调度消息。
from __future__ import print_function
import logging
import logging.handlers
import multiprocessing
import sys
from threading import Thread
class LogMaster(Thread):
QUEUE = None
def __init__(self, file_name):
Thread.__init__(self, name="LoggerThread")
LogMaster.QUEUE = multiprocessing.Queue()
self._file_name = file_name
self._init_logging()
self.start()
def _init_logging(self):
self.logger = logging.getLogger()
self.logger.setLevel(logging.INFO)
fh = logging.FileHandler(self._file_name)
fh.setLevel(logging.INFO)
sh = logging.StreamHandler()
sh.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s\t%(levelname)s\t%(message)s')
fh.setFormatter(formatter)
sh.setFormatter(formatter)
self.logger.addHandler(fh)
self.logger.addHandler(sh)
def _handle_message(self, message):
if message[0] == "print":
print(message[1], end=message[2])
sys.stdout.flush()
elif message[0] == "log":
self.logger.log(message[1], message[2])
def run(self):
while True:
message = LogMaster.QUEUE.get()
if message is None:
break
else:
self._handle_message(message)
def shutdown(self):
LogMaster.QUEUE.put(None)
try:
self.join(2)
except RuntimeError:
pass
@classmethod
def _log(cls, loglevel, msg):
cls.QUEUE.put(("log", loglevel, msg))
@classmethod
def print_term(cls, msg, end='\n'):
cls.QUEUE.put(("print", msg, end))
@staticmethod
def info(msg):
LogMaster._log(logging.INFO, msg)
graph.py
import traceback
from multiprocessing import Process
from logger import LogMaster
class GraphGenerator:
def start_process(self):
LogMaster.print_term("1 - start_process: Print term is working!")
LogMaster.info("1 - start_process: LogMaster.info is working too!")
p = Process(target=self.process_func)
LogMaster.info("starting '{}'".format(self.__class__))
p.start()
# maybe you want to start several processes quickly and rather block
# your main process to wait for them there.
return p
def process_func(self):
LogMaster.print_term("2 - process_func: Print term is working!")
LogMaster.info("2 - process_func: LogMaster.info is working too!")
try:
raise ValueError("checking the exception handling")
except ValueError:
LogMaster.info(traceback.format_exc())
main.py
from graph import GraphGenerator
from logger import LogMaster
if __name__ == '__main__':
SERVICE_NAME = 'MyGraph'
lm = LogMaster(SERVICE_NAME)
LogMaster.info("{} started".format(SERVICE_NAME))
service = GraphGenerator()
child = service.start_process()
# start more child processes if needed
# once all are started, wait for them to complete in the main process
child.join()
LogMaster.info("{} ended".format(SERVICE_NAME))
lm.shutdown()
输出
[shmee@massive test]$ python --version
Python 2.7.5
# ======================================
[shmee@massive test]$ python main.py
2019-03-07 12:28:23,425 INFO MyGraph started
1 - start_process: Print term is working!
2019-03-07 12:28:23,425 INFO 1 - start_process: LogMaster.info is working too!
2019-03-07 12:28:23,425 INFO starting 'graph.GraphGenerator'
2 - process_func: Print term is working!
2019-03-07 12:28:23,426 INFO 2 - process_func: LogMaster.info is working too!
2019-03-07 12:28:23,426 INFO Traceback (most recent call last):
File "/home/shmee/test/graph.py", line 21, in process_func
raise ValueError("checking the exception handling")
ValueError: checking the exception handling
2019-03-07 12:28:23,428 INFO MyGraph ended
# ======================================
[shmee@massive test]$ cat MyGraph
2019-03-07 12:28:23,425 INFO MyGraph started
2019-03-07 12:28:23,425 INFO 1 - start_process: LogMaster.info is working too!
2019-03-07 12:28:23,425 INFO starting 'graph.GraphGenerator'
2019-03-07 12:28:23,426 INFO 2 - process_func: LogMaster.info is working too!
2019-03-07 12:28:23,426 INFO Traceback (most recent call last):
File "/home/shmee/test/graph.py", line 21, in process_func
raise ValueError("checking the exception handling")
ValueError: checking the exception handling
2019-03-07 12:28:23,428 INFO MyGraph ended
# ======================================
# this works as well
[shmee@massive test]$ ../source/python-3.6.4.el7.x86_64/bin/python3 main.py