在新项目中重用Django Celery应用程序，无法找到其他项目模块或celery_app

时间：2018-04-17 17:58:30

标签： django celery django-celery

我有一个使用芹菜来执行后台文件传输和其他数据收集任务（闪烁）的工作应用程序，我想在我的新项目中使用此应用程序的一部分。我已经尝试了几乎所有我能想到的配置排列，但问题仍然存在。从这里开始是运行错误转储：

服务芹菜开始

 Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/site-packages/celery/__main__.py", line 54, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/usr/lib/python3.6/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/usr/lib/python3.6/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 309, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 469, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 489, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/usr/lib/python3.6/site-packages/celery/app/utils.py", line 235, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 492, in symbol_by_name
    return symbol_by_name(name, imp=imp)
  File "/usr/lib/python3.6/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/lib/python3.6/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/cloudscheduler/web_frontend/cloudscheduler/glintwebui/celery_app.py", line 26, in <module>
    django.setup()
  File "/usr/lib64/python3.6/site-packages/django/__init__.py", line 22, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/lib64/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/usr/lib64/python3.6/site-packages/django/conf/__init__.py", line 41, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib64/python3.6/site-packages/django/conf/__init__.py", line 110, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'cloudscheduler_web'
    > w4@csv2.heprc.uvic.ca: * Child terminated with errorcode 1
FAILED

我的服务文件如下：

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=celery
Group=celery
EnvironmentFile=-/etc/cloudscheduler/celery
WorkingDirectory=/opt/cloudscheduler/web_frontend/cloudscheduler/
ExecStart=${CELERY_BIN} multi start $CELERYD_NODES -A \
    $CELERY_APP -logfile=${CELERYD_LOG_FILE} \
    --pidfile=${CELERYD_PID_FILE} $CELERYD_OPTS
ExecStop=${CELERY_BIN} multi stopwait $CELERYD_NODES \
    --pidfile=${CELERYD_PID_FILE}
ExecReload=${CELERY_BIN} multi restart $CELERYD_NODES -A \
    $CELERY_APP --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
    --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS

[Install]
WantedBy=multi-user.target

环境文件：

# Name of nodes to start
# here we have a single node
#CELERYD_NODES="w1"
# or we could have three nodes:
CELERYD_NODES="w1 w2 w3 w4"

# We only want each worker to claim 1 job at a time, especially the worker dedicated to img collection
CELERYD_PREFETCH_MULTIPLIER=1

# Absolute path to "manage.py"
CELERY_BIN="/opt/cloudscheduler/web_frontend/cloudscheduler/manage.py"

# Where to chdir at start. This could be the root of a virtualenv.
CELERYD_CHDIR="/opt/cloudscheduler/web_frontend/cloudscheduler/glintwebui"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="celery_app"
# or fully qualified:
#CELERY_APP="glintwebui:celery_app"


# How to call manage.py
CELERYD_MULTI="celery multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="-Q:w1 image_collection -Q:w2,w3,w4 celery -A celery_app --concurrency=1 -Ofair"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

作为备注，当我将CELERYD_CHDIR更改为顶级项目目录时，它无法找到该应用程序。我为完全限定的CELERY_APP尝试了几个不同的表达式，而chdir设置为顶级项目，但它永远无法找到celery_app。

以下是来自cloudscheduler_web目录中主要settings.pt的相关设置：

CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Canada/Pacific'
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_DEFAULT_QUEUE = 'celery'
CELERY_DEFAULT_EXCHANGE = "celery"
CELERY_QUEUES = {
    "celery": {"exchange": "celery"},
    "image_collection": {"exchange": "image_collection"},
}
CELERY_ROUTES = {
    'cloudscheduler.glintwebui.tasks.image_collection': {'queue': 'image_collection'},
}

最后，这是项目的目录结构：

cloudscheduler
│
├── cloudscheduler_web
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── csv2
│   ├── apps.py
│   ├── cloud_views.py
│   ├── config.py
│   ├── config.pyc
│   ├── csv2_web.yaml
│   ├── group_views.py
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── models.py
│   ├── tests.py
│   ├── urls.py
│   ├── user_views.py
│   ├── views.py
│   └── view_utils.py
├── glintwebui
│   ├── admin.py
│   ├── apps.py
│   ├── celery_app.py
│   ├── config.py
│   ├── forms.py
│   ├── glint_api.py
│   ├── __init__.py
│   ├── models.py
│   ├── tests.py
│   ├── urls.py
│   ├── utils.py
│   ├── __version__.py
│   └── views.py
└── manage.py

我迷失的地方是芹菜如何找到celery_app或其他应用程序模块，但从来没有。我已经多次阅读了芹菜配置文件，我觉得我还是要遗漏一些东西，以便按照原样行事。

如果有帮助的话，我可以发布原始应用程序的目录结构（闪烁）。这里的奖励是celery_app.py文件的一部分：

from __future__ import absolute_import, unicode_literals
import os
import time
import subprocess
import django
from django.conf import settings

from celery import Celery
from celery.utils.log import get_task_logger
#import glintwebui.config as config
import config

from glint_api import repo_connector

logger = get_task_logger(__name__)

# Indicate Celery to use the default Django settings module
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'cloudscheduler_web.settings')

django.setup()

app = Celery('celery_app', broker=config.celery_url, backend=config.celery_backend)
app.config_from_object('django.conf:settings')


@app.task(bind=True)
def debug_task(self):
    logger.debug('Request: {0!r}'.format(self.request))


@app.task(bind=True)
def image_collection(self):

尝试py-D的建议后出错：

Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/celery/app/utils.py", line 241, in find_app
        found = sym.app
    AttributeError: module 'glintwebui' has no attribute 'app'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/usr/lib/python3.6/site-packages/celery/__main__.py", line 54, in <module>
        main()
      File "/usr/lib/python3.6/site-packages/celery/__main__.py", line 30, in main
        main()
      File "/usr/lib/python3.6/site-packages/celery/bin/celery.py", line 81, in main
        cmd.execute_from_commandline(argv)
      File "/usr/lib/python3.6/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
        super(CeleryCommand, self).execute_from_commandline(argv)))
      File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 309, in execute_from_commandline
        argv = self.setup_app_from_commandline(argv)
      File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 469, in setup_app_from_commandline
        self.app = self.find_app(app)
      File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 489, in find_app
        return find_app(app, symbol_by_name=self.symbol_by_name)
      File "/usr/lib/python3.6/site-packages/celery/app/utils.py", line 246, in find_app
        found = sym.celery
    AttributeError: module 'glintwebui' has no attribute 'celery'
        > w4@csv2.heprc.uvic.ca: * Child terminated with errorcode 1
    FAILED

2 个答案:

答案 0 :(得分：0)

在您的情况下，您必须使用app标题glintwebui，而不是celery_app。 只需尝试使用：

app = Celery('glintwebui', broker=config.celery_url, backend=config.celery_backend)

答案 1 :(得分：0)

在尝试了一百万件事后，我终于发现我的 init .py文件缺少芹菜应用程序导入。我想只要您的代码兼容python2-3，就可以在新的python3项目中重用它。

相关问题
最新问题