我正在使用芹菜和弦,并且标题中的一些任务成功,有些失败并出现错误:
celery.backends.base.ChordError: NotRegistered("'app.tasks.grab_articles'",)
但是,当我的Celery工作人员加载时,我看到任务加载如下:
(latentcall)
2017-06-21T22:24:04.505411+00:00 app[worker.1]: ---- **** -----
2017-06-21T22:24:04.505412+00:00 app[worker.1]: --- * *** * -- Linux-3.13.0-121-generic-x86_64-with-debian-stretch-sid 2017-06-21 22:24:04
2017-06-21T22:24:04.505413+00:00 app[worker.1]: -- * - **** ---
2017-06-21T22:24:04.505414+00:00 app[worker.1]: - ** ---------- [config]
2017-06-21T22:24:04.505415+00:00 app[worker.1]: - ** ---------- .> app: app:0x7f3db1959208
2017-06-21T22:24:04.505418+00:00 app[worker.1]: - ** ---------- .> transport: amqp://...
2017-06-21T22:24:04.505419+00:00 app[worker.1]: - ** ---------- .> results: postgres:...
2017-06-21T22:24:04.505420+00:00 app[worker.1]: - *** --- * --- .> concurrency: 1 (prefork)
2017-06-21T22:24:04.505421+00:00 app[worker.1]: -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
2017-06-21T22:24:04.505421+00:00 app[worker.1]: --- ***** -----
2017-06-21T22:24:04.505422+00:00 app[worker.1]: -------------- [queues]
2017-06-21T22:24:04.505423+00:00 app[worker.1]: .> celery exchange=celery(direct) key=celery
2017-06-21T22:24:04.505423+00:00 app[worker.1]:
2017-06-21T22:24:04.505424+00:00 app[worker.1]:
2017-06-21T22:24:04.505425+00:00 app[worker.1]: [tasks]
2017-06-21T22:24:04.505425+00:00 app[worker.1]: . app.tasks.calculate_best_matches
2017-06-21T22:24:04.505426+00:00 app[worker.1]: . app.tasks.grab_articles
我的应用设置与此处的应用相同:
https://github.com/miguelgrinberg/flasky-with-celery
这是我的应用程序的结构:
proj
|---app
|---__init__.py
|---tasks.py
|---celery_worker.py
|---config.py
|---manage.py
__init__.py
from celery import Celery
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from config import Config
db = SQLAlchemy()
celery = Celery(__name__, broker=Config.CELERY_BROKER_URL, backend=Config.CELERY_RESULT_BACKEND)
def create_app():
app = Flask(__name__)
app.config.from_object(Config)
Config.init_app(app)
db.init_app(app)
celery.conf.update(app.config)
from .main import main as main_blueprint
app.register_blueprint(main_blueprint)
return app
tasks.py
from . import celery
from sqlalchemy.sql import select
from sklearn.metrics.pairwise import pairwise_distances
from datetime import datetime
import os
import json
import redis
import pandas as pd
from . import helpers
from . import models
r = redis.from_url(os.environ['REDIS_URL'])
@celery.task(name='task_grab_articles', bind=True)
def grab_articles(self, ids):
print(self.request.id)
article = models.Article
query = article.query.with_entities(article.id, article.tfidf).filter(article.id.in_(ids)).all()
df = pd.DataFrame(query, columns=['id', 'tfidf']).to_json()
return query
celery_worker:
#!/usr/bin/env python
from app import celery, create_app
app = create_app()
app.app_context().push()
manage.py
from app import create_app
app = create_app()
Procfile:
web: gunicorn manage:app
worker: celery worker --app=celery_worker.celery --loglevel=info --concurrency=1
在做了一些调试之后,我发现Celery创建了8个任务,只向我的工作进程发送了三个或四个(参见Procfile),一旦完成这四个任务,我的回调任务将用True响应.ready()。但是,当我检查我的celery_taskmeta表时,其他四个任务仍然是PENDING。除了后端之外,这四个任务在任何地方都不可见。我想知道他们是否会被发送到由于多处理而生成的一些神秘工作者/节点。