Celery worker dyno为单个周期性任务忽略run_every产生多个进程

时间:2015-11-06 00:44:41

标签: python heroku celery celerybeat

我正在建立一个芹菜工人,目的是每n秒运行一次任务。我看到了日志声明。我看到它产生了进程。所以这个方法正在调用。

#portfolio-builder.py

from celery import Celery
from celery.task import periodic_task
from datetime import timedelta

REDIS_URL = "A PROPERLY FORMATTED REDIS URL APPEARED"
app = Celery('tasks')
app.conf.update(BROKER_URL=REDIS_URL,
            CELERY_RESULT_BACKEND=REDIS_URL)


@periodic_task(run_every=timedelta(seconds=30))
def build_portfolios():
    logger = build_portfolios.get_logger()
    logger.info("building portfolios")

我使用procfile运行worker:

worker: celery worker --app=portfolio-builder.app --loglevel=INFO
worker: celery beat --app=portfolio-builder --loglevel=INFO

我运行命令

heroku local

worker.1 | [2015-11-05 18:37:00,279: INFO/MainProcess] beat: Starting...
worker.1 | [2015-11-05 18:37:00,309: WARNING/MainProcess] /Users/user/Documents/Project-Raptor/Raptor-Portfolio-Builder/ENV/lib/python2.7/site-packages/celery/apps/worker.py:161: CDeprecationWarning: 
worker.1 | [2015-11-05 18:37:00,917: INFO/MainProcess] Scheduler: Sending due task portfolio-builder.build_portfolios (portfolio-builder.build_portfolios)
worker.1 | [2015-11-05 18:37:01,215: INFO/MainProcess] Connected to ****//
worker.1 | [2015-11-05 18:37:01,611: INFO/MainProcess] mingle: searching for neighbors
worker.1 | [2015-11-05 18:37:03,016: INFO/MainProcess] mingle: all alone
worker.1 | [2015-11-05 18:37:03,538: WARNING/MainProcess] celery@users-MacBook-Pro.local ready.
worker.1 | [2015-11-05 18:37:03,786: INFO/MainProcess] Received task: portfolio-builder.build_portfolios[8765efc9-f355-4eb9-8afe-385d4f6c6b84]
worker.1 | [2015-11-05 18:37:03,787: INFO/Worker-4] portfolio-builder.build_portfolios[8765efc9-f355-4eb9-8afe-385d4f6c6b84]: building portfolios
worker.1 | [2015-11-05 18:37:03,827: INFO/MainProcess] Task portfolio-builder.build_portfolios[8765efc9-f355-4eb9-8afe-385d4f6c6b84] succeeded in 0.0396841909969s: None
worker.1 | [2015-11-05 18:37:03,867: INFO/MainProcess] Received task: portfolio-builder.build_portfolios[3c6bd54a-2eeb-485d-a737-df752ee93650]
worker.1 | [2015-11-05 18:37:03,869: INFO/Worker-3] portfolio-builder.build_portfolios[3c6bd54a-2eeb-485d-a737-df752ee93650]: building portfolios
worker.1 | [2015-11-05 18:37:03,908: INFO/MainProcess] Task portfolio-builder.build_portfolios[3c6bd54a-2eeb-485d-a737-df752ee93650] succeeded in 0.0399794020341s: None
worker.1 | [2015-11-05 18:37:03,949: INFO/MainProcess] Received task: portfolio-builder.build_portfolios[7d4e8fe9-5dad-42af-b7a8-f6e141fa7835]
worker.1 | [2015-11-05 18:37:03,951: INFO/Worker-2] portfolio-builder.build_portfolios[7d4e8fe9-5dad-42af-b7a8-f6e141fa7835]: building portfolios
worker.1 | [2015-11-05 18:37:03,992: INFO/MainProcess] Task portfolio-builder.build_portfolios[7d4e8fe9-5dad-42af-b7a8-f6e141fa7835] succeeded in 0.0424436000176s: None
worker.1 | [2015-11-05 18:37:04,033: INFO/MainProcess] Received task: portfolio-builder.build_portfolios[848cd495-d063-4e1e-95bf-1636a338659a]
worker.1 | [2015-11-05 18:37:04,035: INFO/Worker-8] portfolio-builder.build_portfolios[848cd495-d063-4e1e-95bf-1636a338659a]: building portfolios
worker.1 | [2015-11-05 18:37:04,074: INFO/MainProcess] Task portfolio-builder.build_portfolios[848cd495-d063-4e1e-95bf-1636a338659a] succeeded in 0.0397574069793s: None
worker.1 | [2015-11-05 18:37:04,115: INFO/MainProcess] Received task: portfolio-builder.build_portfolios[f9adf726-1e26-44ef-ab5e-c05e994553e9]
worker.1 | [2015-11-05 18:37:04,116: INFO/Worker-5] portfolio-builder.build_portfolios[f9adf726-1e26-44ef-ab5e-c05e994553e9]: building portfolios
worker.1 | [2015-11-05 18:37:04,156: INFO/MainProcess] Task portfolio-builder.build_portfolios[f9adf726-1e26-44ef-ab5e-c05e994553e9] succeeded in 0.0404749789741s: None
worker.1 | [2015-11-05 18:37:04,197: INFO/MainProcess] Received task: portfolio-builder.build_portfolios[f994ebae-2860-47ee-98df-db8488d6e01f]
worker.1 | [2015-11-05 18:37:04,199: INFO/Worker-1] portfolio-builder.build_portfolios[f994ebae-2860-47ee-98df-db8488d6e01f]: building portfolios
worker.1 | [2015-11-05 18:37:04,239: INFO/MainProcess] Task portfolio-builder.build_portfolios[f994ebae-2860-47ee-98df-db8488d6e01f] succeeded in 0.0405924500083s: None
worker.1 | [2015-11-05 18:37:04,280: INFO/MainProcess] Received task: portfolio-builder.build_portfolios[27c31f19-7f79-413b-9b0a-e467bd2d0c42]
worker.1 | [2015-11-05 18:37:04,282: INFO/Worker-7] portfolio-builder.build_portfolios[27c31f19-7f79-413b-9b0a-e467bd2d0c42]: building portfolios
worker.1 | [2015-11-05 18:37:04,321: INFO/MainProcess] Task portfolio-builder.build_portfolios[27c31f19-7f79-413b-9b0a-e467bd2d0c42] succeeded in 0.0400763470097s: None
worker.1 | [2015-11-05 18:37:04,362: INFO/MainProcess] Received task: portfolio-builder.build_portfolios[f2c5042c-146e-4b9b-9bc7-a156ef250bf8]
worker.1 | [2015-11-05 18:37:04,363: INFO/Worker-6] portfolio-builder.build_portfolios[f2c5042c-146e-4b9b-9bc7-a156ef250bf8]: building portfolios
worker.1 | [2015-11-05 18:37:04,403: INFO/MainProcess] Task portfolio-builder.build_portfolios[f2c5042c-146e-4b9b-9bc7-a156ef250bf8] succeeded in 0.0400690860115s: None

我的目的是每30秒只在一个工作进程上运行此工作程序。但是在开始时它会产生多个工作进程。另外我打算在Heroku上运行这个,所以如果有任何特别的警告,我会很感激帮助!

1 个答案:

答案 0 :(得分:0)

我弄清楚它是什么!

我实际上已经有一个已经运行的节拍工作但不是正在运行的实际工作队列。因此,大约几个小时,节拍工作者每隔10秒就安排一次任务。

似乎每次我运行它时,工人都会抓取每个计划的任务并立即运行它们,这就是为什么我看到这么多工人。

我还添加了--concurrency 1,这样只有一个工作人员可以运行。