我在我的python项目执行celery任务时遇到一个特殊的问题。我曾尝试 celery.signals和worker_hijack_root_logger = False 来定制我自己的日志，但似乎没有任何效果。如果我能够对其进行格式化，则它似乎有时会打印，而其他时候会跳过日志。这使我想知道我是否缺少成功完成芹菜任务记录的功能。
场景：
我要打印的格式是-
Format=%(version:1.0.0)s %(timestamp:asctime)s %(severity:levelname)s %(service_id:tosca-o)s %(message:message)s
为此，我创建了一个customClass来重写pythonjsonformatter-
from datetime import datetime
from pythonjsonlogger import jsonlogger
class JsonFormatter(jsonlogger.JsonFormatter):
def add_fields(self, log_record, record, message_dict):
"""
Override this method to implement custom logic for adding fields.
"""
record.asctime = self.formatTime(record, self.datefmt)
for field in self._required_fields:
key = field.split(":")[0]
value = field.split(":")[1]
if record.__dict__.get(value):
log_record[key] = record.__dict__.get(value)
else:
log_record[key] = value
log_record.update(message_dict)
jsonlogger.merge_record_extra(record, log_record, reserved=self._skip_fields)
if self.timestamp:
key = self.timestamp if isinstance(self.timestamp, str) else 'timestamp'
log_record[key] = datetime.utcnow()
我的 logging.conf 文件类似于-
[loggers]
keys=root
[handlers]
keys=consoleHandler
[formatters]
keys=customFormatter
[logger_root]
level=DEBUG
handlers=consoleHandler
[handler_consoleHandler]
level=DEBUG
class=StreamHandler
formatter=customFormatter
args=(sys.stdout,)
[formatter_customFormatter]
format=%(version:1.0.0)s %(timestamp:asctime)s %(severity:levelname)s %(service_id:my-service)s %(message:message)s
class=test.myJsonFormatter.JsonFormatter
我有一个 __ init __。py ，在该项目中初始化了项目并设置了记录器-
from flask_httpauth import HTTPBasicAuth
from flask_restful import Resource, Api, reqparse, fields
from flask_restful_swagger import swagger
import celery.events.state
from celery import Celery
from celery.app.log import TaskFormatter
from celery.signals import after_setup_task_logger
from my.ModelClasses import myCommandModel, myModel, myRequestResultModel, myExtraArgsModel
#Set the queue for celery
io_q = Queue()
app = Flask(__name__)
auth = HTTPBasicAuth()
this_path = sys.path[0]
config = SafeConfigParser()
#contains configurations for celery, redis and flask
config.read('config.ini')
#Configure Logging to sys.stdout
logging.config.fileConfig('my/logging.conf')
logger = logging.getLogger(__name__)
app.config['broker_url'] = config.get("Default", "CELERY_BROKER_URL")
app.config['result_backend'] = config.get("Default", "CELERY_RESULT_BACKEND")
str_task_timeout = config.get("Default", "CELERY_TASK_TIMEOUT")
my_root = config.get("Default", "my_root")
my_filter = config.get("Default", "my_filter")
task_timeout = int(str_task_timeout)
api = swagger.docs(Api(app), apiVersion='1.0')
celery = Celery(app.name, broker=app.config['broker_url'], backend=app.config['result_backend'])
celery.control.time_limit('do_long_running_task', soft=900, hard=900, reply=True)
celery.conf.update(app.config)
然后我有一个 celery_task.py ，我要在其中记录执行的任务-
import subprocess
import datetime, time, json
from subprocess import Popen, PIPE
from myapp import api, app, celery, task_timeout, logger
@celery.task(bind=True, soft_time_limit=task_timeout, time_limit=(task_timeout+10))
def do_long_running_task(self, cmd, type='myapp'):
with app.app_context():
output = ""
self.update_state(state='IN_PROGRESS',
meta={'output': output,
'description': "",
'returncode': None})
print(str.format("About to execute: {0}", cmd))
proc = Popen([cmd], stdout=PIPE, stderr=subprocess.STDOUT, shell=True)
logger.info("About to execute %s",cmd)
for line in iter(proc.stdout.readline, b''):
print(str(line))
output = output + line.decode("ascii")
self.update_state(state='IN_PROGRESS', meta={'output': output,'description': "Task is still executing",'returncode': None})
return_code = proc.poll()
if return_code is 0:
meta = {'output': output,
'returncode': proc.returncode,
'description': "Task is complete"
}
self.update_state(state='COMPLETED',
meta=meta)
elif return_code is not 0:
#failure
meta = {'output': output,
'returncode': return_code,
'description': str.format("Celery ran the task, but {0} has reported with error", type)
}
self.update_state(state='FAILED',
meta=meta)
if len(output) is 0:
output = "no output"
meta = {'output': output,
'returncode': return_code,
'description': str.format("Celery ran the task, but {0} has reported with error", type)
}
return meta
上面的 print cmd会打印日志，但不会以我的自定义格式打印日志，即使我将其替换为从 myapp 导入的 logger 对象已根据需要设置记录器。
我尝试使用celery.signals并劫持root，但仍然无法以我的自定义格式打印。
我以-
的形式执行此dockerized服务services:
myService:
image: my-service
ports:
- '8000:8000'
volumes:
- ./vol/data:/data
restart: always
depends_on:
- "celery"
celery:
image: my-service
command: /usr/bin/celery worker -A myapp.celery --loglevel=info
volumes:
- ./vol/data:/data
restart: always
depends_on:
- "redis"
redis:
image: 'redis:alpine'
command: redis-server --appendonly yes
volumes:
- 'redis:/data'
restart: always
ports:
- '6379:6379'
node_alpinex:
image: node_alpinex
ports:
- '4000:22'
volumes:
redis:
因此要执行的celery命令是-
“芹菜工作者-myapp.celery --loglevel = info”
打印示例
celery_1 | [2019-06-21 12:27:18,970: WARNING/ForkPoolWorker-2] b'PLAY [node_alpinex] ************************************************************\n'
celery_1 | [2019-06-21 12:27:18,976: WARNING/ForkPoolWorker-2] b'META: ran handlers\n'
celery_1 | [2019-06-21 12:27:18,994: WARNING/ForkPoolWorker-2] b'\n'
celery_1 | [2019-06-21 12:27:18,997: WARNING/ForkPoolWorker-2] b'TASK [Bootstrap a host without python3 installed] ******************************\n'
celery_1 | [2019-06-21 12:27:18,998: WARNING/ForkPoolWorker-2] b'task path: /usr/src/data/test4.yml:8\n'
我想上面以我的格式打印。有人可以帮忙吗？
文件结构：
app
|--myapp
|--__init__.py
|--celery_task.py
|--logging.conf
|-- etc etc
|--config.ini