Question

帮助！我需要集体意识的力量！就是这样。

我正在尝试为多种服务构建NGINX / Gunicorn / Celery / RabbitMQ / Django服务器。每个都是NGINX / Gunicorn下面的Django应用程序，这很容易。一项服务，让我们称之为“收割机”，有一些芹菜@shared_task和一个celerybeat @periodic_task启动它。另一项服务＆＃39; factory＆＃39;只有一个@shared_task。关键是 - 当celerybeat启动＆＃39; harvester.tasks.harvesting＆＃39;时，它会将结果发送到工厂的API方法中。服务。此方法验证数据，如果正确，则启动工厂＆＃39; task ... like factory_task.delay（record）

这就是问题所在。虽然收割机的两个守护进程都是＆＃39;服务工作正确......工厂的任务是什么？服务根本没有启动。日志中没有记录......没有错误......所有服务都照常运行......

版本：

Ubuntu ：16.04
Python ：2.7.12
Django ：1.10.6
芹菜：4.0.2
Kombu ：4.0.2
RabbitMQ ：3.5.7

在这里配置文件：

** 1。 HARVESTER **

1.1 /etc/systemd/system/celery-harvester.service

[Unit]
Description=celery harvester daemon
After=network.target

[Service]
Type=forking
User=user
Group=group
EnvironmentFile=/var/www/harvester/config/celery.conf
WorkingDirectory=/var/www/harvester/project
# PIDFile=/var/www/harvester/socket/celery_harvester.pid
PrivateTmp=true
Restart=on-failure
NoNewPrivileges=yes
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} -Q ${CELERY_QUEUES} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OP$
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
  --pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} -Q ${CELERY_QUEUES} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OP$

[Install]
WantedBy=multi-user.target

1.2 /var/www/harvester/config/celery.conf

# === CELERY HARVESTER ===

# Name of nodes to start (a single node, "or w1 w2 w3")
CELERYD_NODES="wh"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"

# App instance to use
CELERY_APP="harvester"

# Queues
CELERY_QUEUES="harvester"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--autoscale=8,1"

# PID&LOG
CELERYD_PID_FILE="/var/www/harvester/socket/celery_harvester.pid"
CELERYD_LOG_FILE="/var/www/harvester/log/celery_harvester.log"
CELERYD_LOG_LEVEL="INFO"

# === CELERY BEAT ===

CELERYBEAT_SCHEDULE="/var/www/harvester/socket/harvester-schedule"
CELERYBEAT_PID_FILE="/var/www/harvester/socket/celerybeat_harvester.pid"
CELERYBEAT_LOG_FILE="/var/www/harvester/log/celerybeat_harvester.log"
CELERYBEAT_LOG_LEVEL="INFO"

1.3收割机django芹菜设置

BROKER_URL = 'amqp://user:password@localhost:5672//'
CELERY_TIMEZONE = 'Sol/Moon'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_DEFAULT_QUEUE = 'harvester'
CELERY_QUEUES = (
    Queue('harvester', Exchange('harvester'), routing_key='harvester'),
)
CELERY_ROUTES = {
    'harvester.tasks.parse_rss': {'queue': 'harvester'},
}

** 2。工厂**

2.1 /etc/systemd/system/celery-factory.service

[Unit]
Description=celery factory daemon
After=network.target

[Service]
Type=forking
User=user
Group=group
EnvironmentFile=/var/www/factory/config/celery.conf
WorkingDirectory=/var/www/factory/project
# PIDFile=/var/www/factory/socket/celery_factory.pid
PrivateTmp=true
Restart=on-failure
NoNewPrivileges=yes
ExecStart=/bin/sh -c '${CELERY_BIN} multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} -Q ${CELERY_QUEUES} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OP$
ExecStop=/bin/sh -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \
  --pidfile=${CELERYD_PID_FILE}'
ExecReload=/bin/sh -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} -Q ${CELERY_QUEUES} --pidfile=${CELERYD_PID_FILE} \
  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OP$

[Install]
WantedBy=multi-user.target

2.2 /var/www/factory/config/celery.conf

# === CELERY FACTORY ===

# Name of nodes to start (a single node, "or w1 w2 w3")
CELERYD_NODES="wf"

# Absolute or relative path to the 'celery' command:
CELERY_BIN="/usr/local/bin/celery"

# App instance to use
CELERY_APP="factory"

# Queues
CELERY_QUEUES="factory"

# How to call manage.py
CELERYD_MULTI="multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--autoscale=8,1"

# PID&LOG
CELERYD_PID_FILE="/var/www/factory/socket/celery_factory.pid"
CELERYD_LOG_FILE="/var/www/factory/log/celery_factory.log"
CELERYD_LOG_LEVEL="INFO"

1.3工厂django芹菜设置

BROKER_URL = 'amqp://user:password@localhost:5672//'
CELERY_TIMEZONE = 'Sol/Fobos'
CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_DEFAULT_QUEUE = 'factory'
CELERY_QUEUES = (
    Queue('factory', Exchange('factory'), routing_key='factory'),
)
CELERY_ROUTES = {
    'factory.tasks.parse_record': {'queue': 'factory'},
}

===重要!!! ===

我已经对所有文件权限，用户，群组进行了双重三重检查。它绝对不是RAM或磁盘空间的情况。我发现的唯一的东西 - 有时＆＃39;工厂＆＃39;守护进程工作者将一些打印件写入 / var / log / syslog ，有时写入 journalctl -xe 。但它并没有将任何内容写入其配置中指向的自己的日志文件中。

Answer 1

好的，所以这个问题与Celery中的路由系统有某种关系。也许我以错误的方式配置它，但似乎忽略了Django设置中的 CELERY_DEFAULT_QUEUE ， CELERY_QUEUES 和 CELERY_ROUTES 选项。

对我有用的解决方案是在设置中删除它们，并在 @shared_task 装饰器本身中将它们作为选项提供，如下所示。

@shared_task(default_retry_delay=20, max_retries=5, soft_time_limit=20, queue='factory', routing_key='factory')

另外，我发现Celery工作者的 AUTOSCALE 选项不是一个可靠的选项，因为有时它会导致失去任务，可能是因为它们与某些时间的自动缩放相符。

可在此处找到更多内容 - https://github.com/celery/celery/issues/2682

芹菜系统多种服务一延迟不工作

1 个答案: