Question

我有一个在Linux上成功运行的简单测试代码，但它不能在我的Windows 10 x64计算机上运行。

当我试图开始一个芹菜工人时，它抱怨了不可恢复的错误：PicklingError。（芹菜版：3.1.20）

在我的celery配置中，我将序列化设置为'json'，但它仍然没有任何帮助。

CELERY_RESULT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']

以下是完整的错误消息：

[2016-02-09 15:11:48,532: ERROR/MainProcess] Unrecoverable error: PicklingError("Can't pickle <type 'module'>: it's not found as __builtin__.module",)

Traceback (most recent call last):
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\worker\__init__.py", line 206, in start
    self.blueprint.start(self)   
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 123, in start
    step.start(parent)
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\bootsteps.py", line 374, in start
    return self.obj.start()
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\base.py", line 131, in start
    self.on_start()
  File "C:\Python27\lib\site-packages\celery-3.1.20-py2.7.egg\celery\concurrency\prefork.py", line 117, in on_start
    **self.options)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 972, in __init__
    self._create_worker_process(i)
  File "C:\Python27\lib\site-packages\billiard\pool.py", line 1068, in _create_worker_process
    w.start()
  File "C:\Python27\lib\site-packages\billiard\process.py", line 137, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\site-packages\billiard\forking.py", line 263, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\site-packages\billiard\py2\reduction.py", line 84, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 562, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 396, in save_reduce
    save(cls)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 748, in save_global
    (obj, module, name))
PicklingError: Can't pickle <type 'module'>: it's not found as __builtin__.module

Answer 1

我遇到了同样的问题。奇怪的是，这个问题只存在于Windows上，在Linux Celery运行时没有任何问题。结果我需要将配置模块作为名称而不是对象传递：

而不是

app.config_from_object('celeryconfig')

来自Celery docs的说明：

提示

建议使用模块名称，因为这意味着在使用prefork池时不需要序列化模块。如果您遇到配置pickle错误，请尝试使用模块名称。

显然，当Celery作为在Windows上失败的对象传递时，它需要挑选配置。如果作为模块名称传递它是有效的。谢谢你指点我正确的方向@JoyLy！

Answer 2

@jeverling's answer的补充。
如果您想监视配置文件的目标位置，可以使用__name__变量。

from . import celery_config

celery_app.config_from_object(celery_config.__name__)

使用“PicklingError”无法在Windows 10上启动Celery worker

2 个答案:

提示