Question

我希望使用Celery / RabbitMQ在Windows 7和Python 3.3上的Flask应用程序中运行regualr interval任务。 Pip安装：

billiard==3.3.0.10
celery==3.1.6

代码：

### celeryapp.py ###
from celery import Celery
from twend import config, app

def make_celery(app):
    celery = Celery(app.import_name, broker=app.config['CELERY_BROKER_URL'])
    celery.conf.update(app.config)
    TaskBase = celery.Task
    class ContextTask(TaskBase):
        abstract = True
        def __call__(self, *args, **kwargs):
            with app.app_context():
                return TaskBase.__call__(self, *args, **kwargs)
    celery.Task = ContextTask
    return celery

celery_pipe = make_celery(app)
celery_pipe.config_from_object(config)

### tasks.py ###
from celery import task
from twend.celeryapp import celery_pipe as celery

@celery.task()
def add_together(a, b):
    return a + b

### config.py ###
CELERY_BROKER_URL = 'amqp://'
CELERY_RESULT_BACKEND = 'amqp://'
CELERY_ACCEPT_CONTENT = ['json']

当我尝试启动工作人员时，我得到了：

ERROR/MainProcess] Unrecoverable error: PicklingError(
"Can't pickle <class 'module'>: attribute lookup builtins.module failed",)
Traceback (most recent call last):
  File "c:\Python33\twend\lib\site-packages\celery\worker\__init__.py", line 212
, in start
    self.blueprint.start(self)
  File "c:\Python33\twend\lib\site-packages\celery\bootsteps.py", line 123, in s
tart
    step.start(parent)
  File "c:\Python33\twend\lib\site-packages\celery\bootsteps.py", line 373, in s
tart
    return self.obj.start()
  File "c:\Python33\twend\lib\site-packages\celery\concurrency\base.py", line 12
7, in start
    self.on_start()
  File "c:\Python33\twend\lib\site-packages\celery\concurrency\prefork.py", line
 112, in on_start
    **self.options)
  File "c:\Python33\twend\lib\site-packages\billiard\pool.py", line 966, in __in
it__
    self._create_worker_process(i)
  File "c:\Python33\twend\lib\site-packages\billiard\pool.py", line 1059, in _cr
eate_worker_process
    w.start()
  File "c:\Python33\twend\lib\site-packages\billiard\process.py", line 139, in s
tart
    self._popen = Popen(self)
  File "c:\Python33\twend\lib\site-packages\billiard\forking.py", line 263, in _
_init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "c:\Python33\twend\lib\site-packages\billiard\_reduction3.py", line 60, i
n dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'module'>: attribute lookup builtins.
module failed

(twend) c:\Python33\twend>Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "c:\Python33\twend\lib\site-packages\billiard\forking.py", line 457, in m
ain
    self = load(from_parent)
EOFError

这https://github.com/celery/django-celery/issues/228似乎是一个类似的问题，但在我正在使用的Celery版本中修复了。

我还读到Windows对酸洗有限制，但我不确定究竟是什么腌制。

Answer 1

Celery使用酸洗来序列化任务和结果。我看到你已经将你的配置设置为只接受json序列化，所以也许你可以通过增加几个配置行来解决你的问题：

CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'

这样，根本不会使用pickle，所有Celery的序列化都将使用json完成。理论上这可能会导致性能下降，但我在Flask应用程序中以这种方式解决了类似的问题，并没有发现任何性能问题。 YMMV

在Flask应用程序中开始使用Celery时的PicklingError

1 个答案: