Question

我需要使用Celery在一组主机上安排Fabric任务。

我的代码是：

tasks.py

from fabric_tasks import poll
from api.client import APIClient as client
from celery import Celery
celery = Celery(broker='redis://')

@celery.task
def poll_all():
  actions = client().get_actions(status='SCHEDULED)
  ids = [a['id'] for a in actions]
  execute(poll, ids, hosts=hosts)

CELERYBEAT_SCHEDULE = {
    'every-5-second': {
        'task': 'tasks.poll_all',
        'schedule': timedelta(seconds=5),
    },
}

芹菜-A tasks.celery -B -l info 任务失败，出现以下异常：

[2016-12-11 17:54:07,928: ERROR/PoolWorker-3] Task tasks.poll_actions[7b26a083-b450-4a90-8971-97f3dd2f4f5d] raised unexpected: AttributeError("'Process' object has no attribute '_authkey'",)
Traceback (most recent call last):
  File "/Users/ocervell/.virtualenvs/ndc-v3.3/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/Users/ocervell/.virtualenvs/ndc-v3.3/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
    return self.run(*args, **kwargs)
  File "/Users/ocervell/Drive/workspace/ccc/svn/build-support/branches/development-ocervell/modules/ndc/v3.3/tasks.py", line 44, in poll_all
    return execute(poll, 'ls', hosts=hosts)
  File "/Users/ocervell/.virtualenvs/ndc-v3.3/lib/python2.7/site-packages/fabric/tasks.py", line 387, in execute
    multiprocessing
  File "/Users/ocervell/.virtualenvs/ndc-v3.3/lib/python2.7/site-packages/fabric/tasks.py", line 269, in _execute
    p = multiprocessing.Process(target=inner, kwargs=kwarg_dict)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 98, in __init__
    self._authkey = _current_process._authkey
AttributeError: 'Process' object has no attribute '_authkey'

临时修复

tasks.py 中的以下hack帮助我修复了这个错误（实际上它设置了导致AttributeError异常的属性）：

from celery.signals import worker_process_init

@worker_process_init.connect
def fix_multiprocessing(**kwargs):
    from multiprocessing import current_process
    keys = {
      '_authkey': '',
      '_daemonic': False,
      '_tempdir': ''
    }
    for k, v in keys.items():
        try:
            getattr(current_process(), k)
        except AttributeError:
            setattr(current_process(), k, v)

相关：

https://github.com/celery/celery/issues/1709

Python multiprocessing job to Celery task but AttributeError

替代品？

我想知道Celery是否有更好的方法来安排Fabric任务而不在multiprocessing库周围写脏东西？

使用Celery运行Fabric任务

0 个答案: