Question

如何使用django和Elastic Benastalk，它也只能在主节点上通过celery运行任务？

Answer 1

这就是我在弹性beanstalk上使用 django 设置芹菜的方法，可扩展性正常。

请注意 container_commands 的'leader_only'选项仅适用于环境重建或部署要处理该问题，您可能必须为领导节点应用实例保护。检查：http://docs.aws.amazon.com/autoscaling/latest/userguide/as-instance-termination.html#instance-protection-instance

为芹菜工作者添加bash脚本并节拍配置。

添加文件 root_folder / .ebextensions / files / celery_configuration.txt ：

#!/usr/bin/env bash # Get django environment variables celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'` celeryenv=${celeryenv%?} # Create celery configuraiton script celeryconf="[program:celeryd-worker] ; Set full path to celery program if using virtualenv command=/opt/python/run/venv/bin/celery worker -A django_app --loglevel=INFO directory=/opt/python/current/app user=nobody numprocs=1 stdout_logfile=/var/log/celery-worker.log stderr_logfile=/var/log/celery-worker.log autostart=true autorestart=true startsecs=10 ; Need to wait for currently executing tasks to finish at shutdown. ; Increase this if you have very long running tasks. stopwaitsecs = 600 ; When resorting to send SIGKILL to the program to terminate it ; send SIGKILL to its whole process group instead, ; taking care of its children as well. killasgroup=true ; if rabbitmq is supervised, set its priority higher ; so it starts first priority=998 environment=$celeryenv [program:celeryd-beat] ; Set full path to celery program if using virtualenv command=/opt/python/run/venv/bin/celery beat -A django_app --loglevel=INFO --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid directory=/opt/python/current/app user=nobody numprocs=1 stdout_logfile=/var/log/celery-beat.log stderr_logfile=/var/log/celery-beat.log autostart=true autorestart=true startsecs=10 ; Need to wait for currently executing tasks to finish at shutdown. ; Increase this if you have very long running tasks. stopwaitsecs = 600 ; When resorting to send SIGKILL to the program to terminate it ; send SIGKILL to its whole process group instead, ; taking care of its children as well. killasgroup=true ; if rabbitmq is supervised, set its priority higher ; so it starts first priority=998 environment=$celeryenv" # Create the celery supervisord conf script echo "$celeryconf" | tee /opt/python/etc/celery.conf # Add configuration script to supervisord conf (if not there already) if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf then echo "[include]" | tee -a /opt/python/etc/supervisord.conf echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf fi # Reread the supervisord config supervisorctl -c /opt/python/etc/supervisord.conf reread # Update supervisord in cache without restarting all services supervisorctl -c /opt/python/etc/supervisord.conf update # Start/Restart celeryd through supervisord supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker

在部署期间注意脚本执行，但仅限于主节点（leader_only：true）。添加文件 root_folder / .ebextensions / 02-python.config ：

container_commands: 04_celery_tasks: command: "cat .ebextensions/files/celery_configuration.txt > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh" leader_only: true 05_celery_tasks_run: command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh" leader_only: true

Beat是可配置的，无需重新部署，具有单独的django应用程序：https://pypi.python.org/pypi/django_celery_beat。

存储任务结果最好：https://pypi.python.org/pypi/django_celery_beat

档案 requirements.txt

celery==4.0.0 django_celery_beat==1.0.1 django_celery_results==1.0.1 pycurl==7.43.0 --global-option="--with-nss"

为Amazon SQS代理配置celery （从列表中获取所需的端点：http://docs.aws.amazon.com/general/latest/gr/rande.html）的 ROOT_FOLDER / django_app / settings.py ：

... CELERY_RESULT_BACKEND = 'django-db' CELERY_BROKER_URL = 'sqs://%s:%s@' % (aws_access_key_id, aws_secret_access_key) # Due to error on lib region N Virginia is used temporarily. please set it on Ireland "eu-west-1" after fix. CELERY_BROKER_TRANSPORT_OPTIONS = { "region": "eu-west-1", 'queue_name_prefix': 'django_app-%s-' % os.environ.get('APP_ENV', 'dev'), 'visibility_timeout': 360, 'polling_interval': 1 } ...

django django_app app
的Celery配置
添加文件 root_folder / django_app / celery.py ：

from __future__ import absolute_import, unicode_literals import os from celery import Celery # set the default Django settings module for the 'celery' program. os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'django_app.settings') app = Celery('django_app') # Using a string here means the worker don't have to serialize # the configuration object to child processes. # - namespace='CELERY' means all celery-related configuration keys # should have a `CELERY_` prefix. app.config_from_object('django.conf:settings', namespace='CELERY') # Load task modules from all registered Django app configs. app.autodiscover_tasks()

修改文件 root_folder / django_app / __ init __。py ：

from __future__ import absolute_import, unicode_literals # This will make sure the app is always imported when # Django starts so that shared_task will use this app. from django_app.celery import app as celery_app __all__ = ['celery_app']

检查：

How do you run a worker with AWS Elastic Beanstalk?（无可扩展性的解决方案）

Pip Requirements.txt --global-option causing installation errors with other packages. "option not recognized"（来自弹性beanstalk的绝对点的问题的解决方案，无法处理正确解决pycurl依赖的全局选项）

Answer 2

这就是我通过@smentek扩展答案的方式，以允许多个工作人员实例和一个节拍实例–在必须保护领导者的地方也有同样的事情。（我现在还没有自动化的解决方案。）

请注意，通过EB cli或Web界面对EB进行envvar更新不会受到芹菜节拍或工作人员的影响，直到重新启动应用程序服务器为止。这一次让我措手不及。

单个celery_configuration.sh文件输出两个脚本供监管者使用，请注意celery-beat具有autostart=false，否则在实例重新启动后您将获得许多节拍：

# get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g' | sed 's/%/%%/g'`
celeryenv=${celeryenv%?}

# create celery beat config script
celerybeatconf="[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A lexvoco --loglevel=INFO --workdir=/tmp -S django --pidfile /tmp/celerybeat.pid

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=false
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 10

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv"

# create celery worker config script
celeryworkerconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A lexvoco --loglevel=INFO

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=999

environment=$celeryenv"

# create files for the scripts
echo "$celerybeatconf" | tee /opt/python/etc/celerybeat.conf
echo "$celeryworkerconf" | tee /opt/python/etc/celeryworker.conf

# add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
  then
  echo "[include]" | tee -a /opt/python/etc/supervisord.conf
  echo "files: celerybeat.conf celeryworker.conf" | tee -a /opt/python/etc/supervisord.conf
fi

# reread the supervisord config
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf reread
# update supervisord in cache without restarting all services
/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf update

然后在container_commands中，我们仅在领导者身上重新开始节拍：

container_commands:
  # create the celery configuration file
  01_create_celery_beat_configuration_file:
    command: "cat .ebextensions/files/celery_configuration.sh > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && sed -i 's/\r$//' /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
  # restart celery beat if leader
  02_start_celery_beat:
    command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat"
    leader_only: true
  # restart celery worker
  03_start_celery_worker:
    command: "/usr/local/bin/supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker"

Answer 3

如果有人遵循smentek的回答并得到错误：

grid.draw(gp)

知道，如果您使用Windows，则问题可能是“ celery_configuration.txt”文件在具有UNIX EOL时具有WINDOWS EOL。如果使用Notepad ++，请打开文件，然后单击“编辑> EOL转换> Unix（LF）”。保存，重新部署和错误不再存在。

另外，对像我这样的真正业余人群的一些警告：

确保在settings.py文件的“ INSTALLED_APPS”中包含“ django_celery_beat”和“ django_celery_results”。
要检查celery错误，请使用“ eb ssh”连接到您的实例，然后使用“ tail -n 40 /var/log/celery-worker.log”和“ tail -n 40 / var / log / celery”连接到您的实例-beat.log”（其中“ 40”表示要从文件末尾开始读取的行数）。

希望这对某人有帮助，它将为我节省一些时间！

如何使用AWS Elastic Beanstalk可扩展的Django app运行芹菜工作者？

3 个答案: