我正在尝试使用Django 1.9.8,Celery 4.0.2,RabbitMQ 2.1.4,Redis 2.10.5安排每10分钟运行一次的任务。这些都在Linux(Fedora 25)中的Docker容器中运行。我尝试了很多我在Celery文档和本网站上找到的东西组合。到目前为止,唯一有效的组合如下。但是,它仅在应用程序启动时最初运行周期性任务,但此后将忽略该计划。我已经完全确认计划任务在初始时间后没有再次运行。
我的(几乎工作)设置只运行一次:
INSTALLED_APPS = (
...
'django_celery_beat',
...
)
BROKER_URL = 'amqp://{user}:{password}@{hostname}/{vhost}/'.format(
user=os.environ['RABBIT_USER'],
password=os.environ['RABBIT_PASS'],
hostname=RABBIT_HOSTNAME,
vhost=os.environ.get('RABBIT_ENV_VHOST', '')
# We don't want to have dead connections stored on rabbitmq, so we have to negotiate using heartbeats
BROKER_HEARTBEAT = '?heartbeat=30'
if not BROKER_URL.endswith(BROKER_HEARTBEAT):
BROKER_URL += BROKER_HEARTBEAT
BROKER_POOL_LIMIT = 1
BROKER_CONNECTION_TIMEOUT = 10
# Celery configuration
# configure queues, currently we have only one
CELERY_DEFAULT_QUEUE = 'default'
CELERY_QUEUES = (
Queue('default', Exchange('default'), routing_key='default'),
)
# Sensible settings for celery
CELERY_ALWAYS_EAGER = False
CELERY_ACKS_LATE = True
CELERY_TASK_PUBLISH_RETRY = True
CELERY_DISABLE_RATE_LIMITS = False
# By default we will ignore result
# If you want to see results and try out tasks interactively, change it to False
# Or change this setting on tasks level
CELERY_IGNORE_RESULT = True
CELERY_SEND_TASK_ERROR_EMAILS = False
CELERY_TASK_RESULT_EXPIRES = 600
# Set redis as celery result backend
CELERY_RESULT_BACKEND = 'redis://%s:%d/%d' % (REDIS_HOST, REDIS_PORT, REDIS_DB)
CELERY_REDIS_MAX_CONNECTIONS = 1
# Don't use pickle as serializer, json is much safer
CELERY_TASK_SERIALIZER = "json"
CELERY_RESULT_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ['application/json']
CELERYD_HIJACK_ROOT_LOGGER = False
CELERYD_PREFETCH_MULTIPLIER = 1
CELERYD_MAX_TASKS_PER_CHILD = 1000
coding=UTF8
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "web_portal.settings")
app = Celery('web_portal')
CELERY_TIMEZONE = 'UTC'
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
from celery.schedules import crontab
from .celeryconf import app as celery_app
@celery_app.on_after_finalize.connect
def setup_periodic_tasks(sender, **kwargs):
# Calls email_scanner every 10 minutes
sender.add_periodic_task(
crontab(hour='*',
minute='*/10',
second='*',
day_of_week='*',
day_of_month='*'),
email_scanner.delay(),
)
@app.task
def email_scanner():
dispatch_list = scanning.email_scan()
for dispatch in dispatch_list:
validate_dispatch.delay(dispatch)
return
#!/bin/sh
# wait for RabbitMQ server to start
sleep 10
cd web_portal
# run Celery worker for our project myproject with Celery configuration stored in Celeryconf
su -m myuser -c "celery beat -l info --pidfile=/tmp/celerybeat-web_portal.pid -s /tmp/celerybeat-schedule &"
su -m myuser -c "celery worker -A web_portal.celeryconf -Q default -n default@%h"
我还尝试在settings.py中使用CELERYBEAT_SCHEDULER代替@ celery_app.on_after finalize_connect装饰器并在tasks.py中阻止,但调度程序从未运行过一次。
(与以前相同,但也包括以下内容)
CELERYBEAT_SCHEDULE = {
'email-scanner-every-5-minutes': {
'task': 'tasks.email_scanner',
'schedule': timedelta(minutes=10)
},
}
在线Celery 4.0.2文档假定我应该本能地知道许多赠品,但我在这种环境中是新手。如果有人知道我在哪里可以找到除docs.celeryproject.org和http://django-celery-beat.readthedocs.io/en/latest/之外的教程,他们都认为我已经是Django大师,我将不胜感激。或者让我知道,如果你在我的设置中看到一些明显错误的东西。谢谢!
答案 0 :(得分:0)
我找到了一个有效的解决方案。我无法让CELERYBEAT_SCHEDULE或芹菜任务装饰工作,我怀疑它可能至少部分归因于我开始芹菜击败任务的方式。
工作解决方案全程9码,以利用Django Database Scheduler。我下载了GitHub项目" https://github.com/celery/django-celery-beat"并将所有代码合并为另一个" app"在我的项目中。这使Django-Admin访问能够通过浏览器维护cron / interval / periodic任务表。我还修改了我的run_celery.sh,如下所示:
#!/bin/sh
# wait for RabbitMQ server to start
sleep 10
# run Celery worker for our project myproject with Celery configuration stored in Celeryconf
celery beat -A web_portal.celeryconf -l info --pidfile=/tmp/celerybeat- web_portal.pid -S django --detach
su -m myuser -c "celery worker -A web_portal.celeryconf -Q default -n default@%h -l info "
通过django-admin Web界面添加计划任务后,调度程序开始正常工作。