Question

我有一组5个格式为.npz的文件。我需要逐个从这些文件中提取numpy数组，然后用它来训练模型。在内存中加载第一个numpy数组并用它训练模型后，如果我尝试通过切片将其从内存中删除，消耗的内存不会减少。因此，我无法加载第二个numpy数组并最终得到MemoryError。

如何在训练模型后确保释放内存？

PS：X_test和y_test的大小非常小，因此可以忽略。

代码：

for person_id in range(1, 5):
     print "Initial memory ",resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
     temp1 = np.load("../final_data/speaker_input" + str(person_id))
     X_train = temp1['arr_0']
     y_train = np.load("../final_data/speaker_final_output" + str(person_id)+ ".npy")
     X_test,y_test = data.Test(person_id=1)
     print "Input dimension ", X_train.shape
     print "Output dimension",y_train.shape
     print "Before training ",resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
     lipreadtrain.train(model=net,X_train=X_train, y_train=y_train,X_test=X_test, y_test=y_test)
     print "After training ", resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
     X_train =  X_train[:1]
     y_train = y_train[:1]
     X_test = X_test[:1]
     y_test = y_test[:1]
     print len(X_train),len(y_train),len(X_test),len(y_test)
     gc.collect()
     temp1.close()

输出：

Initial memory  861116
Input dimension  (8024, 50, 2800)
Output dimension (8024, 53)
Before training  9642152
Training the model, which will take a long long time...
Epoch 1/1
8024/8024 [==============================] - 42s - loss: nan - acc: 0.2316        
----- Training Takes 42.3187870979 Seconds -----
Finished!
After training  9868080
1 1 0 0
Initial memory  9868080
Traceback (most recent call last):
File "test.py", line 21, in <module>
X_train = temp1['arr_0']
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/numpy/lib/npyio.py",           line 224, in __getitem__
pickle_kwargs=self.pickle_kwargs)
File "/home/ubuntu/anaconda2/lib/python2.7/site-packages/numpy/lib/format.py", line 661, in read_array
array = numpy.empty(count, dtype=dtype)
MemoryError

Answer 1

问题在于切片只是一个视图，但它会阻止底层内存，甚至你删除或重命名父名称。举例说明：

from pylab import *
res=[]
i=0
while True:
    t=zeros(1e10,uint8)
    u=t[:1]
    res.append(u)
    i=i+1
    print(i)

给出：

In [17]: (executing lines 1 to 9 of "<tmp 1>")
1
2
Traceback (most recent call last):
  File "<tmp 1>", line 5, in <module>
    t=zeros(1e10,uint8)
MemoryError

现在，只需制作副本：u=t[:1].copy()而不是u=t[:1]，然后在每个循环中释放t：

In [18]: (executing lines 1 to 9 of "<tmp 1>")
1
2
3
4
5
....

无法释放numpy数组所消耗的内存

1 个答案: