使用Celery + Django无法在task.py中导入模型

时间:2018-07-19 15:44:22

标签: python django celery python-import

我想创建一个后台任务来更新特定日期的记录。我将Django和Celery与RabbitMQ结合使用。

使用此虚拟任务函数保存模型时,我设法使任务被调用:

tasks.py

from __future__ import absolute_import
from celery import Celery
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)

app = Celery('tasks', broker='amqp://localhost//')


@app.task(name='news.tasks.update_news_status')
def update_news_status(news_id):
    # (I pass the news id and return it, nothing complicated about it)
    return news_id

此任务是通过 models.py

中的save()方法调用的
from django.db import models
from celery import current_app


class News(models.model):
    (...)

    def save(self, *args, **kwargs):

        current_app.send_task('news.tasks.update_news_status', args=(self.id,))

        super(News, self).save(*args, **kwargs)

事情是我想将我的新闻模型导入tasks.py ,但是如果我想这样:

from .models import News

我收到此错误:

  

django.core.exceptions.ImproperlyConfigured:请求的设置   DEFAULT_INDEX_TABLESPACE,但未配置设置。你必须   定义环境变量DJANGO_SETTINGS_MODULE或调用   settings.configure(),然后再访问设置。

mi celery.py 的外观如下

from __future__ import absolute_import, unicode_literals
from celery import Celery
import os
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings')
app = Celery('myapp')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

我已经尝试过:

  1. can't import django model into celery task
  2. 我试图在任务方法Django and Celery, AppRegisteredNotReady exception内进行导入
  3. 我也尝试过这个Celery - importing models in tasks.py
  4. 我还尝试创建一个utils.py并将其导入,这是不可能的。

并遇到了不同的错误,但是最后,我无法在task.py

中导入任何模块。

我的配置可能有问题,但是我看不到错误,我按照The Celery Docs: First steps with Django

中的步骤进行操作

此外,我的项目结构如下:

├── myapp
│   ├── __init__.py
├── ├── celery.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── news
│   ├── __init__.py
│   ├── admin.py
│   ├── apps.py
│   ├── tasks.py
│   ├── urls.py
│   ├── models.py
│   ├── views.py
├── manage.py

我正在像这样从myapp目录执行工作程序:

celery -A news.tasks worker --loglevel=info

我在这里想念什么?预先感谢您的帮助!

lambda:settings.INSTALLED_APPS

编辑

进行注释中建议的更改后: 将此添加到celery.py app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

并导入内部方法:tasks.py

from __future__ import absolute_import
from celery import Celery
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)

app = Celery('tasks', broker='amqp://localhost//')


@app.task(name='news.tasks.update_news_status')
def update_news_status(news_id):
    from .models import News
    return news_id

我收到以下错误:

[2018-07-20 12:24:29,337: ERROR/ForkPoolWorker-1] Task news.tasks.update_news_status[87f9ec92-c260-4ee9-a3bc-5f684c819f79] raised unexpected: ValueError('Attempted relative import in non-package',)
Traceback (most recent call last):
  File "/Users/carla/Develop/App/backend/myapp-venv/lib/python2.7/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Users/carla/Develop/App/backend/myapp-venv/lib/python2.7/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/carla/Develop/App/backend/news/tasks.py", line 12, in update_news_status
    from .models import News
ValueError: Attempted relative import in non-package

3 个答案:

答案 0 :(得分:2)

好吧,所以对于任何为此苦苦挣扎的人来说……原来我的celery.py并不是从设置中读取环境变量。

经过一周的研究,我意识到 Celery不是Django的进程,而是在Django之外运行的进程(duh),所以当我尝试加载设置时,它们被加载了,但是那么我将无法访问在.env中定义的env变量(我使用dotenv库)。 Celery试图在我的.bash_profile中查找env变量(当然)

最后,我的解决方案是在定义了celery.py的同一目录中创建一个帮助器模块,并使用以下

来命名为load_env.py
from os.path import dirname, join
import dotenv


def load_env():
    "Get the path to the .env file and load it."
    project_dir = dirname(dirname(__file__))
    dotenv.read_dotenv(join(project_dir, '.env'))

然后在我的 celery.py 上(注意最后的导入和第一条说明)

from __future__ import absolute_import, unicode_literals
from celery import Celery
from django.conf import settings
import os
from .load_env import load_env

load_env()

# set the default Django settings module for the 'celery' program.
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")

app = Celery('myapp')

# Using a string here means the worker doesn't have to serialize
# the configuration object to child processes.
# - namespace='CELERY' means all celery-related configuration keys
#   should have a `CELERY_` prefix.

app.config_from_object('myapp.settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

在加载对load_env()环境变量的调用之后,芹菜工作者可以访问它们。这样,我现在可以从我的task.py 中访问其他模块,这是我的主要问题。

贷记this guys (Caktus Consulting Group)及其django-project-template,因为如果不是他们的话,我将找不到答案。谢谢。

答案 1 :(得分:0)

尝试类似这样的方法。它在3.1芹菜中工作,导入应在save方法内并在super()之后进行

from django.db import models



class News(models.model):
    (...)

    def save(self, *args, **kwargs):
        (...)
        super(News, self).save(*args, **kwargs)
        from task import update_news_status
        update_news_status.apply_async((self.id,)) #apply_async or delay

答案 2 :(得分:0)

这是我要执行的操作(Django 1.11和celery 4.2),您的celery配置中有问题,您尝试重新声明Celery实例:

tasks.py

from myapp.celery import app # would contain what you need :)
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)

@app.task(name='news.tasks.update_news_status')
def update_news_status(news_id):
    # (I pass the news id and return it, nothing complicated about it)
    return news_id

celery.py

from __future__ import absolute_import, unicode_literals
from celery import Celery
from django.conf import settings
import os

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")
app = Celery('myapp', backend='rpc://', broker=BROKER_URL) # your config here
app.config_from_object('django.myapp:settings', namespace='CELERY') # change here
app.autodiscover_tasks()

models.py

from django.db import models

class News(models.model):
    (...)
    def save(self, *args, **kwargs):
        super(News, self).save(*args, **kwargs)
        from news.tasks import update_news_status
        update_news_status.delay(self.id) # change here

并使用celery -A myapp worker --loglevel=info启动它,因为您的应用程序是在myapp.celery中定义的,因此-A参数必须是声明conf的应用程序