Question

从pickle加载multiprocessing.manager.dictionary时遇到错误。我确信pickle文件存在于我运行python的目录中。

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> import multiprocessing
>>> a = open("test.pkl", "rb")
>>> pickle.load(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce
    value = func(*args)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 879, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 733, in __init__
    self._incref()
  File "/usr/lib/python2.7/multiprocessing/managers.py", line 783, in _incref
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
  File "/usr/lib/python2.7/multiprocessing/connection.py", line 304, in SocketClient
    s.connect(address)
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
socket.error: [Errno 2] No such file or directory
>>> pickle.load(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1165, in load_put
    self.memo[self.readline()[:-1]] = self.stack[-1]
IndexError: list index out of range

以下是将其保存为泡菜的代码部分。

from multiprocessing import Manager, Pool
import pickle
from functools import partial

def output(dic, s):
    a = open(s + ".pkl", "wb")
    pickle.dump(dic, a)
    a.close()

data_list = [1,2,3,4]
pool = Pool(processes = 4)
m = Manager()
lock = m.Lock()
dic1 = m.dict()
func = partial(f, dic1) # f is a function that takes 3 arguments with the returned result stored in dic1
pool.map(func, data_list)
output(dic1, "test")
pool.close()
pool.join()

如果我在输出之前打印字典，终端上的结果看起来很好。

Answer 1

Manager.dict()实际上并没有将dict而是代理对象返回给管理员在不同进程中管理的真实字典。当您在该对象上调用方法时，该调用将转发到实际字典所在的管理器。

由于管理器和客户端进程之间的通信是使用pickle协议进行的，因此这些对象是可选择的，并且当unickled为再次生成管理器进程内的实际对象生成代理时，这就是为什么你看到尝试当你试图破坏字典时创建一个连接。

因此，如果你想挑选经理dict的内容，请在之前将其转换为实际的dict：

...
pickle.dump(dict(dic), a)
...

从pickle文件加载multiprocessing.manager.dictionary时出错

1 个答案: