Question

以下是给我“内存错误”的片段。当计数器达到约53009525时。我在Ubuntu虚拟机上运行它，内存为12GB。

from collections import defaultdict
from collections import OrderedDict
import math
import numpy as np
import operator
import time
from itertools import product

class State():
   def __init__(self, field1, field2, field3, field4, field5, field6):
       self.lat = field1
       self.lon = field2
       self.alt = field3
       self.temp = field4
       self.ws = field5
       self.wd = field6
...
trans = defaultdict(dict)
freq = {}
...
matrix_col = {}
matrix_col = OrderedDict(sorted(freq.items(), key=lambda t: t[0].lat,    reverse=True))

trans_mat = []
counter = 0 
for u1, u2 in product(matrix_col, matrix_col):
    print counter, time.ctime()
    if (u1 in trans) and (u2 in trans[u1]):
        trans_mat.append([trans[u1][u2]*1.0])
    else:
        trans_mat.append([[0.0001]])
    counter += 1

trans_mat = np.asarray(trans_mat)
trans_mat = np.reshape(trans_mat, (10734, 10734))
print trans_mat

freq 和 trans 都会存储类型＆＃34;状态＆＃34;。蚂蚁帮助表示赞赏。这是错误： ...

53009525 Mon Oct 12 18:11:16 2015
Traceback (most recent call last):
  File "hmm_freq.py", line 295, in <module>
     trans_mat.append([[0.0001]])
MemoryError

Answer 1

看起来好像在每次迭代时，你都会在两个嵌套列表中附加一个Python float到trans_mat。您可以使用sys.getsizeof检查每个元素的大小：

import sys

# each of the two nested lists is 80 bytes
print(sys.getsizeof([]))
# 80

# a Python float object is 24 bytes
print(sys.getsizeof(0.0001))
# 24

在每次迭代中，您将2 * 80 + 24 = 184个字节附加到trans_mat。在53009525次迭代后，您将附加9753752600字节或9.75GB。

一种使内存效率更高的简单方法是将结果直接存储到numpy数组而不是嵌套列表中：

trans_mat = np.empty(10734 * 10734, np.double)
counter = 0
for u1, u2 in product(matrix_col, matrix_col):
    print counter, time.ctime()
    if (u1 in trans) and (u2 in trans[u1]):

        # you may need to check that this line yields a float
        # (it's hard for me to tell exactly what `trans` is from your code snippet)
        trans_mat[counter] = trans[u1][u2]*1.0

    else:
        trans_mat[counter] = 0.0001
    counter += 1

供参考：

# the size of the Python container is 80 bytes
print(sys.getsizeof(trans_mat))
# 80

# the size of the array buffer is 921750048 bytes or 921 MB
print(trans_mat.nbytes)
# 921750048

Numpy Array Memory错误，12GB RAM

1 个答案: