Django celerybeat周期性任务只运行一次

时间:2017-02-20 20:56:42

标签: python django rabbitmq celerybeat

我正在尝试使用Django 1.9.8,Celery 4.0.2,RabbitMQ 2.1.4,Redis 2.10.5安排每10分钟运行一次的任务。这些都在Linux(Fedora 25)中的Docker容器中运行。我尝试了很多我在Celery文档和本网站上找到的东西组合。到目前为止,唯一有效的组合如下。但是,它仅在应用程序启动时最初运行周期性任务,但此后将忽略该计划。我已经完全确认计划任务在初始时间后没有再次运行。

我的(几乎工作)设置只运行一次:

settings.py:

INSTALLED_APPS = (
   ...
   'django_celery_beat',
   ...
)
BROKER_URL = 'amqp://{user}:{password}@{hostname}/{vhost}/'.format(
    user=os.environ['RABBIT_USER'],
    password=os.environ['RABBIT_PASS'],
    hostname=RABBIT_HOSTNAME,
    vhost=os.environ.get('RABBIT_ENV_VHOST', '')

# We don't want to have dead connections stored on rabbitmq, so we have to negotiate using heartbeats
BROKER_HEARTBEAT = '?heartbeat=30'
if not BROKER_URL.endswith(BROKER_HEARTBEAT):
    BROKER_URL += BROKER_HEARTBEAT

BROKER_POOL_LIMIT = 1
BROKER_CONNECTION_TIMEOUT = 10

# Celery configuration

# configure queues, currently we have only one
CELERY_DEFAULT_QUEUE = 'default'
CELERY_QUEUES = (
    Queue('default', Exchange('default'), routing_key='default'),
)

# Sensible settings for celery
CELERY_ALWAYS_EAGER = False
CELERY_ACKS_LATE = True
CELERY_TASK_PUBLISH_RETRY = True
CELERY_DISABLE_RATE_LIMITS = False

# By default we will ignore result
# If you want to see results and try out tasks interactively, change it to False
# Or change this setting on tasks level
CELERY_IGNORE_RESULT = True
CELERY_SEND_TASK_ERROR_EMAILS = False
CELERY_TASK_RESULT_EXPIRES = 600

# Set redis as celery result backend
CELERY_RESULT_BACKEND = 'redis://%s:%d/%d' % (REDIS_HOST, REDIS_PORT, REDIS_DB)
CELERY_REDIS_MAX_CONNECTIONS = 1

# Don't use pickle as serializer, json is much safer
CELERY_TASK_SERIALIZER = "json"
CELERY_RESULT_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ['application/json']
CELERYD_HIJACK_ROOT_LOGGER = False
CELERYD_PREFETCH_MULTIPLIER = 1
CELERYD_MAX_TASKS_PER_CHILD = 1000

celeryconf.py

coding=UTF8
from __future__ import absolute_import
import os
from celery import Celery
from django.conf import settings

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "web_portal.settings")
app = Celery('web_portal')
CELERY_TIMEZONE = 'UTC'
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

tasks.py

from celery.schedules import crontab
from .celeryconf import app as celery_app

@celery_app.on_after_finalize.connect
def setup_periodic_tasks(sender, **kwargs):
    # Calls email_scanner every 10 minutes
    sender.add_periodic_task(
        crontab(hour='*', 
                minute='*/10', 
                second='*', 
                day_of_week='*',  
                day_of_month='*'),
        email_scanner.delay(),
    )

@app.task
def email_scanner():
    dispatch_list = scanning.email_scan()
    for dispatch in dispatch_list:
        validate_dispatch.delay(dispatch)
    return

run_celery.sh - 用于从docker-compose.yml启动celery任务

#!/bin/sh

# wait for RabbitMQ server to start
sleep 10

cd web_portal
# run Celery worker for our project myproject with Celery configuration stored in Celeryconf
su -m myuser -c "celery beat -l info --pidfile=/tmp/celerybeat-web_portal.pid -s /tmp/celerybeat-schedule &"
su -m myuser -c "celery worker -A web_portal.celeryconf -Q default -n default@%h"

我还尝试在settings.py中使用CELERYBEAT_SCHEDULER代替@ celery_app.on_after finalize_connect装饰器并在tasks.py中阻止,但调度程序从未运行过一次。

settings.py(不适用于所有情况)

(与以前相同,但也包括以下内容)

CELERYBEAT_SCHEDULE = {
    'email-scanner-every-5-minutes': {
        'task': 'tasks.email_scanner',
        'schedule': timedelta(minutes=10)
    },
}

在线Celery 4.0.2文档假定我应该本能地知道许多赠品,但我在这种环境中是新手。如果有人知道我在哪里可以找到除docs.celeryproject.org和http://django-celery-beat.readthedocs.io/en/latest/之外的教程,他们都认为我已经是Django大师,我将不胜感激。或者让我知道,如果你在我的设置中看到一些明显错误的东西。谢谢!

1 个答案:

答案 0 :(得分:0)

我找到了一个有效的解决方案。我无法让CELERYBEAT_SCHEDULE或芹菜任务装饰工作,我怀疑它可能至少部分归因于我开始芹菜击败任务的方式。

工作解决方案全程9码,以利用Django Database Scheduler。我下载了GitHub项目" https://github.com/celery/django-celery-beat"并将所有代码合并为另一个" app"在我的项目中。这使Django-Admin访问能够通过浏览器维护cron / interval / periodic任务表。我还修改了我的run_celery.sh,如下所示:

#!/bin/sh

# wait for RabbitMQ server to start
sleep 10
# run Celery worker for our project myproject with Celery configuration stored in Celeryconf
celery beat -A web_portal.celeryconf -l info --pidfile=/tmp/celerybeat- web_portal.pid -S django --detach
su -m myuser -c "celery worker -A web_portal.celeryconf -Q default -n default@%h -l info "

通过django-admin Web界面添加计划任务后,调度程序开始正常工作。