dask.protocol.core中的msgpack序列化引发TypeError:无法序列化“变量”对象

时间:2020-07-23 16:14:44

标签: python dask dask-distributed msgpack

我已经分发了dask,这是对工作流程(完美)的更大努力的一部分,并且在我的dask工作者中看到了一个错误。工人收到了工作,但很快提出了例外情况:

TypeError: can not serialize 'Variable' object

完整跟踪为:

distributed.protocol.core - CRITICAL - Failed to Serialize
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 25, in dumps
    small_header, small_payload = dumps_msgpack(msg)
  File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 163, in dumps_msgpack
    payload = msgpack.dumps(msg, use_bin_type=True)
  File "/opt/conda/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Variable' object
distributed.comm.utils - INFO - Unserializable Message: {'op': 'variable_get', 'timeout': 0, 'name': <distributed.variable.Variable object at 0x7fec9eafaef0>, 'client': 'Client-worker-365168fe-ccf9-11ea-800b-0242ac110004', 'reply': True}
distributed.comm.utils - ERROR - can not serialize 'Variable' object
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/distributed/comm/utils.py", line 35, in _to_frames
    msg, serializers=serializers, on_error=on_error, context=context
  File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 25, in dumps
    small_header, small_payload = dumps_msgpack(msg)
  File "/opt/conda/lib/python3.7/site-packages/distributed/protocol/core.py", line 163, in dumps_msgpack
    payload = msgpack.dumps(msg, use_bin_type=True)
  File "/opt/conda/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Variable' object

我已经将其归结为一个非常小的测试用例,我认为这是相当有代表性的。它引起了同样的问题:

from distributed.protocol.core import dumps_msgpack
from distributed.variable import Variable
from distributed.client import Client
client = Client(address='10.72.112.29:38786')
var = Variable(name='test', client=client)
msg = {'op': 'variable_get', 'timeout': 0, 'name': var, 'client': client.id, 'reply': True}
d = dumps_msgpack(msg)

此示例会产生错误消息,但也会显示一些版本不匹配(我不认为是问题所在):

/Users/gdesmarais/git/rightsize/rightsize-venv/lib/python3.7/site-packages/distributed/client.py:1136: VersionMismatchWarning: Mismatched versions found
+---------+---------------+---------------+---------------+
| Package | client        | scheduler     | workers       |
+---------+---------------+---------------+---------------+
| numpy   | 1.19.0        | 1.18.1        | 1.18.1        |
| python  | 3.7.8.final.0 | 3.7.0.final.0 | 3.7.0.final.0 |
+---------+---------------+---------------+---------------+
  warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
Traceback (most recent call last):
  File "<input>", line 7, in <module>
  File "/Users/gdesmarais/git/rightsize/rightsize-venv/lib/python3.7/site-packages/distributed/protocol/core.py", line 163, in dumps_msgpack
    payload = msgpack.dumps(msg, use_bin_type=True)
  File "/Users/gdesmarais/git/rightsize/rightsize-venv/lib/python3.7/site-packages/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 286, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 292, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 289, in msgpack._cmsgpack.Packer.pack
  File "msgpack/_packer.pyx", line 225, in msgpack._cmsgpack.Packer._pack
  File "msgpack/_packer.pyx", line 283, in msgpack._cmsgpack.Packer._pack
TypeError: can not serialize 'Variable' object

我缺少包裹吗?我可以提供安装脚本等,但是涉及到一些脚本-客户端是本地python env,调度程序和工作程序是docker。

1 个答案:

答案 0 :(得分:0)

此问题通过{@ {3}}通过PR:Prefect 0.12.5(针对后代)得到解决。