我正在尝试使用python中提供的'dump'命令将字典转换为pickle格式。字典的文件大小约为150 MB,但只有115 MB的文件被转储时才会出现异常。例外是:
Traceback (most recent call last):
File "C:\Python27\generate_traffic_pattern.py", line 32, in <module>
b.dump_data(way_id_data,'way_id_data.pickle')
File "C:\Python27\class_dump_load_data.py", line 8, in dump_data
pickle.dump(data,saved_file)
File "C:\Python27\lib\pickle.py", line 1370, in dump
Pickler(file, protocol).dump(obj)
File "C:\Python27\lib\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "C:\Python27\lib\pickle.py", line 663, in _batch_setitems
save(v)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 600, in save_list
self._batch_appends(iter(obj))
File "C:\Python27\lib\pickle.py", line 615, in _batch_appends
save(x)
File "C:\Python27\lib\pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "C:\Python27\lib\pickle.py", line 599, in save_list
self.memoize(obj)
File "C:\Python27\lib\pickle.py", line 247, in memoize
self.memo[id(obj)] = memo_len, obj
MemoryError
我真的很困惑,因为我的相同代码工作得很好。
答案 0 :(得分:2)
你只是倾销那一个对象,而这就是全部吗?
如果您多次调用dump,那么在转储之间调用Pickler.clear_memo()
将刷新内部存储的反向引用(导致'泄漏')。你的代码应该可以正常工作......
答案 1 :(得分:2)
你试过这个吗?
import cPickle as pickle
p = pickle.Pickler(open("temp.p","wb"))
p.fast = True
p.dump(d) # d is your dictionary