我一直在Django项目中使用celery（python中的异步任务队列），Redis作为结果后端和代理几年了 - 今天我尝试将后端/代理切换到AWS的ElastiCache Redis，但是celery worker进入无限的启动和关闭周期，如下面日志文件中所示。我还成功地从运行芹菜的同一台机器连接到Redis实例，并成功完成了一些基本的SET和GET操作。
我目前的设置是：
celery worker --concurrency=3 --queues=general --events --loglevel=DEBUG --beat --schedule=/tmp/celerybeat-schedule --pidfile=/tmp/celery.main.pid --config=celeryconfig
import os
from django.conf import settings
from celery import Celery, Task, chain
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings')
celery = Celery()
celery.config_from_object("celeryconfig")
celery.autodiscover_tasks(settings.INSTALLED_APPS, related_name='tasks')
BROKER_URL = "redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0"
CELERY_TIMEZONE = u'America/Los_Angeles'
CELERYD_HIJACK_ROOT_LOGGER = True
CELERY_IMPORTS = (
"app.tasks"
)
CELERYD_TASK_TIME_LIMIT = 35 # seconds
CELERYD_TASK_SOFT_TIME_LIMIT = 25 # seconds
DJANGO_SETTINGS_MODULE = "settings"
CELERY_RESULT_BACKEND = BROKER_URL
CELERY_ROUTES = {
'app.tasks.my_task': {'queue': 'general'}
}
-------------- celery@ip-10-20-21-9 v3.1.24 (Cipater)
---- **** -----
--- * *** * -- Linux-3.13.0-74-generic-x86_64-with-Ubuntu-14.04-trusty
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: default:0x7f7dc6d71650 (.default.Loader)
- ** ---------- .> transport: redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
- ** ---------- .> results: redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
- *** --- * --- .> concurrency: 3 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> general exchange=general(direct) key=general
[tasks]
. app.tasks.my_task
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
[2017-06-16 14:33:12,658: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
[2017-06-16 14:33:15,110: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2017-06-16 14:33:15,112: DEBUG/MainProcess] | Worker: Building graph...
[2017-06-16 14:33:15,113: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Queues (intra), Pool, Autoscaler, StateDB, Autoreloader, Beat, Consumer}
[2017-06-16 14:33:15,118: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2017-06-16 14:33:15,118: DEBUG/MainProcess] | Consumer: Building graph...
[2017-06-16 14:33:15,127: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Heart, Agent, Gossip, event loop}
[2017-06-16 14:33:15,137: DEBUG/MainProcess] | Worker: Starting Hub
[2017-06-16 14:33:15,137: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,137: DEBUG/MainProcess] | Worker: Starting Pool
[2017-06-16 14:33:15,287: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,288: DEBUG/MainProcess] | Worker: Starting Consumer
[2017-06-16 14:33:15,288: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-06-16 14:33:15,315: INFO/MainProcess] Connected to redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
[2017-06-16 14:33:15,315: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,316: DEBUG/MainProcess] | Consumer: Starting Events
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Hub...
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Pool...
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Consumer...
[2017-06-16 14:33:15,329: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2017-06-16 14:33:15,329: DEBUG/MainProcess] | Worker: Stopping Pool...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Worker: Stopping Hub...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2017-06-16 14:33:16,340: DEBUG/MainProcess] | Consumer: Shutdown Connection...
-------------- celery@ip-10-20-21-9 v3.1.24 (Cipater)
---- **** -----
--- * *** * -- Linux-3.13.0-74-generic-x86_64-with-Ubuntu-14.04-trusty
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: default:0x7fdc3b019650 (.default.Loader)
- ** ---------- .> transport: redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
- ** ---------- .> results: redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
- *** --- * --- .> concurrency: 3 (prefork)
-- ******* ----
--- ***** ----- [queues]
-------------- .> general exchange=general(direct) key=general
[tasks]
. app.tasks.my_task
. celery.backend_cleanup
. celery.chain
. celery.chord
. celery.chord_unlock
. celery.chunks
. celery.group
. celery.map
. celery.starmap
[2017-06-16 14:33:12,658: DEBUG/MainProcess] removing tasks from inqueue until task handler finished
[2017-06-16 14:33:15,110: DEBUG/MainProcess] | Worker: Preparing bootsteps.
[2017-06-16 14:33:15,112: DEBUG/MainProcess] | Worker: Building graph...
[2017-06-16 14:33:15,113: DEBUG/MainProcess] | Worker: New boot order: {Timer, Hub, Queues (intra), Pool, Autoscaler, StateDB, Autoreloader, Beat, Consumer}
[2017-06-16 14:33:15,118: DEBUG/MainProcess] | Consumer: Preparing bootsteps.
[2017-06-16 14:33:15,118: DEBUG/MainProcess] | Consumer: Building graph...
[2017-06-16 14:33:15,127: DEBUG/MainProcess] | Consumer: New boot order: {Connection, Events, Mingle, Tasks, Control, Heart, Agent, Gossip, event loop}
[2017-06-16 14:33:15,137: DEBUG/MainProcess] | Worker: Starting Hub
[2017-06-16 14:33:15,137: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,137: DEBUG/MainProcess] | Worker: Starting Pool
[2017-06-16 14:33:15,287: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,288: DEBUG/MainProcess] | Worker: Starting Consumer
[2017-06-16 14:33:15,288: DEBUG/MainProcess] | Consumer: Starting Connection
[2017-06-16 14:33:15,315: INFO/MainProcess] Connected to redis://staging-redis.abcde.clustercfg.use1.cache.amazonaws.com:6379/0
[2017-06-16 14:33:15,315: DEBUG/MainProcess] ^-- substep ok
[2017-06-16 14:33:15,316: DEBUG/MainProcess] | Consumer: Starting Events
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Hub...
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Pool...
[2017-06-16 14:33:15,328: DEBUG/MainProcess] | Worker: Closing Consumer...
[2017-06-16 14:33:15,329: DEBUG/MainProcess] | Worker: Stopping Consumer...
[2017-06-16 14:33:15,329: DEBUG/MainProcess] | Worker: Stopping Pool...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Worker: Stopping Hub...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Consumer: Shutdown Gossip...
[2017-06-16 14:33:16,338: DEBUG/MainProcess] | Consumer: Shutdown Heart...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Control...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Tasks...
[2017-06-16 14:33:16,339: DEBUG/MainProcess] | Consumer: Shutdown Events...
[2017-06-16 14:33:16,340: DEBUG/MainProcess] | Consumer: Shutdown Connection...
## And again...
答案 0 :(得分：0)
看起来问题出现是因为我试图使用Redis群集作为开箱即用的代理和后端 - 在更基本的测试设置中，SETNX命令在芹菜上失败启动，因为看起来Redis集群不支持该命令。
我仍然需要做一些研究，看看是否可以将Redis群集用作Celery后端和代理。