Question

我有芹菜的django项目。我看到一些工人不止一次接受具有相同ID的任务的问题。这是一个日志片段：

9238 [2015-07-03 10:25:03,310: INFO/MainProcess] Task ssrweb.execution.tasks.get_script_results[72171e05-665b-4730-8499-3da821838935] succeeded in 0.3505859375s: None
9239 [2015-07-03 10:25:03,310: INFO/MainProcess] Got task from broker: ssrweb.execution.tasks.get_script_results[70d6a7d3-3cb4-4e01-8d04-4d084e36bdfb]
9240 [2015-07-03 10:25:03,444: INFO/MainProcess] Task ssrweb.execution.tasks.get_script_results[08b6bb3f-6a8a-43a1-992d-e16ceae76e1d] succeeded in 0.750570058823s: None
9241 [2015-07-03 10:25:03,445: INFO/MainProcess] Got task from broker: ssrweb.execution.tasks.get_script_results[8df19680-ba43-4a43-8316-cfbb9bb95e30]
9242 [2015-07-03 10:25:03,633: INFO/MainProcess] Task ssrweb.execution.tasks.get_script_results[72171e05-665b-4730-8499-3da821838935] succeeded in 0.30401802063s: None
9243 [2015-07-03 10:25:03,639: INFO/MainProcess] Got task from broker: ssrweb.execution.tasks.get_script_results[70d6a7d3-3cb4-4e01-8d04-4d084e36bdfb]
9244 [2015-07-03 10:25:03,746: INFO/MainProcess] Task ssrweb.execution.tasks.get_script_results[8df19680-ba43-4a43-8316-cfbb9bb95e30] succeeded in 0.298280000687s: None
9245 [2015-07-03 10:25:03,747: INFO/MainProcess] Got task from broker: ssrweb.execution.tasks.get_script_results[8df19680-ba43-4a43-8316-cfbb9bb95e30]
9246 [2015-07-03 10:25:03,894: INFO/MainProcess] Task ssrweb.execution.tasks.get_script_results[70d6a7d3-3cb4-4e01-8d04-4d084e36bdfb] succeeded in 0.563042879105s: None
9247 [2015-07-03 10:25:03,895: INFO/MainProcess] Got task from broker: ssrweb.execution.tasks.get_script_results[3b2d1317-4ff1-4279-84ae-696d35997358]
9248 [2015-07-03 10:25:04,120: INFO/MainProcess] Task ssrweb.execution.tasks.get_script_results[8df19680-ba43-4a43-8316-cfbb9bb95e30] succeeded in 0.332364082336s: None
`enter code here`9249 [2015-07-03 10:25:04,122: INFO/MainProcess] Got task from broker: ssrweb.execution.tasks.get_script_results[3b2d1317-4ff1-4279-84ae-696d35997358]
9250 [2015-07-03 10:25:04,218: INFO/MainProcess] Task ssrweb.execution.tasks.get_script_results[70d6a7d3-3cb4-4e01-8d04-4d084e36bdfb] succeeded in 0.550192832947s: None
9251 [2015-07-03 10:25:04,219: INFO/MainProcess] Got task from broker: ssrweb.execution.tasks.get_script_results[e7854ef6-6dbf-4840-87bb-699e32d168ce]

见第7239,9243行重复的id 70d6a7d3-3cb4-4e01-8d04-4d084e36bdfb，并在第9246和9250行执行。

这些任务在django管理页面???中显示为问号。

我正在使用（我知道这些是旧版本）：

host:230> ./manage.py celeryd --version
1.3.1
celery 2.5.3
django-celery 2.4.2

这就是我开始工作和celerybeat的方式：

1 #!/bin/sh
2 
3 # not used
4 #./manage.py celeryd -l info -E -B --scheduler=djcelery.schedulers.DatabaseScheduler
5 #./manage.py celeryd -l info -E -B --scheduler=djcelery.schedulers.DatabaseScheduler --pidfile=/opt/ssr/var/run/celery/celery.pid
6 
7 CELERY_LOG_DIR="/opt/ssr/var/log/celery"
8 CELERY_RUN_DIR="/opt/ssr/var/run/celery"
9 CELERY_SCHEDULER="djcelery.schedulers.DatabaseScheduler"
10 LANG=en_US.UTF8
11 
12 mkdir -p $CELERY_LOG_DIR
13 mkdir -p $CELERY_RUN_DIR
14 
15 # Workers for periodic tasks in execution and bvt
16 ./manage.py celeryd_multi start w1 -c 3 -l info -E -Q p_action --logfile=$CELERY_LOG_DIR/w1.log --pidfile=$CELERY_RUN_DIR/w1.pid
17 ./manage.py celeryd_multi start w2 -c 3 -l info -E -Q p_action,p_check,p_cleanup,p_notify --logfile=$CELERY_LOG_DIR/w2.log --pidfile=$CELERY_RUN_DIR/w2.pid
18 
19 # Workers for ondemand tasks in execution and bvt
20 ./manage.py celeryd_multi start w3 -c 3 -l info -E -Q od_action --logfile=$CELERY_LOG_DIR/w3.log --pidfile=$CELERY_RUN_DIR/w3.pid
21 ./manage.py celeryd_multi start w4 -c 3 -l info -E -Q od_action,od_notify,od_check --logfile=$CELERY_LOG_DIR/w4.log --pidfile=$CELERY_RUN_DIR/w4.pid
22 
23 # Workers for the default queue
24 ./manage.py celeryd_multi start w5 -c 2 -l info -E -Q default --logfile=$CELERY_LOG_DIR/w5.log --pidfile=$CELERY_RUN_DIR/w5.pid
25 ./manage.py celeryd_multi start w6 -c 2 -l info -E -Q default --logfile=$CELERY_LOG_DIR/w6.log --pidfile=$CELERY_RUN_DIR/w6.pid
26 
27 # Celerybeat
28 ./manage.py celerybeat -S $CELERY_SCHEDULER

打开/关闭此设置无效：

#CELERY_ACKS_LATE = True
#CELERYD_PREFETCH_MULTIPLIER = 1

我已经看到问题here，但还没有看到最终解决方案。

我也看过this one，但它也没有决定性的解决方案，我想提供我的确切背景。

任何帮助都将不胜感激。

Answer 1

这也发生在我身上在我的tasks.py我有两个periodic_tasks，其中一个比另一个花费的时间更长，最初开始。

@periodic_tasks(run_every=timedelta(minutes=15))
def start_monitoring()  ## This was taking some time to get started.

@periodic_tasks(run_every=timedelta(minutes=5))
def if_script_no_run_then_run_script()  ## This runs quickly but ended up spawning multiple scripts when spinning up the workers.

我最终把速度更快periodic_tasks放在慢速之前，这对我有用。

芹菜工人不止一次接受具有相同id的任务

1 个答案: