Question

当我的一个单元测试删除一个SQLAlchemy对象时，该对象会触发一个after_delete事件，该事件触发Celery任务从驱动器中删除文件。

测试时任务为CELERY_ALWAYS_EAGER = True。

该示例有两个测试。一个触发事件中的任务，另一个触发事件。只有事件中的那个关闭连接。

要快速重现您可以运行的错误：

git clone https://gist.github.com/5762792fc1d628843697.git
cd 5762792fc1d628843697
virtualenv venv
. venv/bin/activate
pip install -r requirements.txt
python test.py

筹码：

$     python test.py
E
======================================================================
ERROR: test_delete_task (__main__.CeleryTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test.py", line 73, in test_delete_task
    db.session.commit()
  File "/home/brice/Code/5762792fc1d628843697/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/scoping.py", line 150, in do
    return getattr(self.registry(), name)(*args, **kwargs)
  File "/home/brice/Code/5762792fc1d628843697/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 776, in commit
    self.transaction.commit()
  File "/home/brice/Code/5762792fc1d628843697/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 377, in commit
    self._prepare_impl()
  File "/home/brice/Code/5762792fc1d628843697/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 357, in _prepare_impl
    self.session.flush()
  File "/home/brice/Code/5762792fc1d628843697/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1919, in flush
    self._flush(objects)
  File "/home/brice/Code/5762792fc1d628843697/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/brice/Code/5762792fc1d628843697/venv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 63, in __exit__
    compat.reraise(type_, value, traceback)
  File "/home/brice/Code/5762792fc1d628843697/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 2037, in _flush
    transaction.rollback(_capture_exception=True)
  File "/home/brice/Code/5762792fc1d628843697/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 393, in rollback
    self._assert_active(prepared_ok=True, rollback_ok=True)
  File "/home/brice/Code/5762792fc1d628843697/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 223, in _assert_active
    raise sa_exc.ResourceClosedError(closed_msg)
ResourceClosedError: This transaction is closed

----------------------------------------------------------------------
Ran 1 test in 0.014s

FAILED (errors=1)

Answer 1

我认为我发现了问题 - 它是如何设置Celery任务的。如果从芹菜设置中删除应用程序上下文调用，一切运行正常：

class ContextTask(TaskBase):
    abstract = True

    def __call__(self, *args, **kwargs):
        # deleted --> with app.app_context():
        return TaskBase.__call__(self, *args, **kwargs)

在SQLAlchemy文档中有一个很大的警告，即在after_delete事件期间永远不会修改会话：http://docs.sqlalchemy.org/en/latest/orm/events.html#sqlalchemy.orm.events.MapperEvents.after_delete

所以我怀疑在删除期间调用了with app.app_context():，试图附加和/或修改Flask-SQLAlchemy存储在app对象中的会话，因此整个事情就是轰炸。

Flask-SQlAlchemy为你做了很多魔术，但你可以绕过这个并直接使用SQLAlchemy。如果您需要在删除事件期间与数据库通信，则可以创建到db的新会话：

@celery.task()
def my_task():
    # obviously here I create a new object
    session = db.create_scoped_session()
    session.add(User(id=13, value="random string"))
    session.commit()
    return

但听起来你不需要这个，你只是想删除一个图像路径。在这种情况下，我只会更改您的任务，因此它需要一条路径：

# instance will call the task
@event.listens_for(User, "after_delete")
def after_delete(mapper, connection, target):
    my_task.delay(target.value)

@celery.task()
def my_task(image_path):
    os.remove(image_path)

希望这有用 - 请告诉我这些内容是否对你有用。感谢非常详细的设置，它确实有助于调试。

Answer 2

类似于deBrice提出的答案，但使用类似于Rachel的方法。

class ContextTask(TaskBase):
    abstract = True

    def __call__(self, *args, **kwargs):
        import flask
        # tests will be run in unittest app context
        if flask.current_app:
            return TaskBase.__call__(self, *args, **kwargs)
        else:
            # actual workers need to enter worker app context 
            with app.app_context():
                return TaskBase.__call__(self, *args, **kwargs)

Answer 3

芹菜的创造者Ask建议解决方案on github

from celery import signals

def make_celery(app):
     ...

     @signals.task_prerun.connect
     def add_task_flask_context(sender, **kwargs):
         if not sender.request.is_eager:
            sender.request.flask_context = app.app_context().__enter__()

    @signals.task_postrun.connect
    def cleanup_task_flask_context(sender, **kwargs):
       flask_context = getattr(sender.request, 'flask_context', None)
       if flask_context is not None:
           flask_context.__exit__(None, None, None)

当SQLAlchemy事件触发Celery任务时，将关闭连接

3 个答案: