保存9GB数据帧时出现泡菜内存错误

时间:2018-12-30 19:31:17

标签: python pickle

我正在将大约100个文件夹中的图像读取到数据框中,每个文件夹在数据框中都有一行。阅读这些图像后,我想将它们保存到pkl文件中。当我使用

 pd.to_pickle(dataset,'.\\dataset.pkl')

正在显示内存错误,我尝试使用hdf5,parquest之类的不同方法,但仍显示error。以下是数据框信息:

<class 'pandas.core.frame.DataFrame'>
Index: 60 entries, dev_01 to dev_09
Data columns (total 1 columns):
videos    60 non-null object
dtypes: object(1)
memory usage: 9.7 GB


 videos_dataframe

folder_01   [[[0.0, 0.0, 0.0, 0.0, 0.0, 34.0, 49.0, 60.0, ...
folder_010  [[[0.0, 0.0, 0.0, 66.0, 75.0, 73.0, 73.0, 73.0...
folder_011  [[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0...
folder_012  [[[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0...
folder_013  [[[0.0, 0.0, 0.0, 28.0, 41.0, 58.0, 73.0, 81.0...
folder_014  [[[0.0, 0.0, 0.0, 0.0, 0.0, 15.0, 26.0, 48.0, ...
folder_015   [[[0.0, 0.0, 0.0, 0.0, 9.0, 8.0, 8.0, 16.0, 30...

我将数据框分为10部分:

splits=np.array_split(df, 10)
with open('split_pickle.pkl', 'wb') as pickle_out:  
pickle.dump(splits, pickle_out)

但仍然显示内存错误:

          Traceback (most recent call last)
         <ipython-input-23-51fab4cebf2c> in <module> 
         3 with open('dev_split_pickle.pkl', 'wb') as pickle_out:
       ----> 4     pickle.dump(splits, pickle_out)

 MemoryError: 

0 个答案:

没有答案