Question

我是numpy的新手，但仍然无法理解它如何处理内存。我知道我的代码在内存方面并不是最优的，但如果不了解内存的处理方式，就无法想到其他方法。这是代码：

# X.shape = (700, 23000)
# Y.shape = (700, 23000)
learning_rate=0.7
iters=100

W = np.random.random((Y.shape[0], X.shape[0])) - 0.5
num_examples = Y.shape[1]

for i in range(int(iters)):
    error = W.dot(X) - Y
    xerror = X.dot(error.T)
    d = (1.0 / num_training_examples * xerror).T
    W = W - (learning_rate * d)

当训练样例的数量很少（~1000）时，代码可以工作，但是当它大到20K时，内存会爆炸。任何有助于优化此代码的内存占用的帮助都非常受欢迎。

Answer 1

It depends on what you mean by the memory "explodes". I believe numpy uses double for matrices and vectors. That means 8 bytes per number. Therefore a 20k x 20k problem is going to be 400M numbers or 3.2 GB. If that's what you mean then it's just a problem of scale (you have too big of a problem) and you need to find a different way to represent it if such memory uses are too large.

Based on your comment about X and Y being matrices. If your problem size is only 20k ish you can get some savings by only handling a column at a time of X and Y. At 23k rows that will reduce your memory footprint to 1/3. If you scale up to 46k rows that will be up to 2/3 (only a reduction of 1/3) and by 92k rows you'll only reduce to 16/18 (0.888).

Generally when you go to large problems you start dealing with things like iterative algorithms and sparse matrices to reduce the memory load. See for instance conjugate gradient.

如何减少这个numpy梯度下降代码的内存占用？

1 个答案: