Question

我正在研究具有非常大的稀疏权重矩阵的神经网络。权重中的零值保持为零，不能更改。只应为非零值计算和传播梯度，否则运行起来会太昂贵。这意味着我正在使用Theano及其sparse.structured_dot矩阵乘法。这是我的代码：

#!/usr/bin/env python3
import math
import numpy as np
import theano as th
import theano.tensor as T
from theano import sparse as sp
import scipy.sparse as spsp


def sparse_matrix(matrix, name):
    """Convert an array to a shared sparse theano matrix."""
    matrix = np.asarray(matrix)
    if matrix.shape[0] > matrix.shape[1]: matrix = spsp.csc_matrix(matrix)
    else: matrix = spsp.csr_matrix(matrix)
    return th.shared(matrix, name)


def mul(weight, matrix):
    """Sparse matrix multiplication.
    :param weight: sparse weight m x n matrix, where m is number of features of output and n is number of features for 
    input.
    :param matrix: input n x p matrix, where n is number of features for input and p is number of input vectors.
    """
    return T.transpose(sp.structured_dot(weight, T.transpose(matrix)))


x = T.fmatrix('x')
target = T.fmatrix('target')

W = sparse_matrix([[0, 0.5], [0.5, 0]], 'W')
y = mul(W, x)

cost = T.mean((y - target) ** 2) / 2
gradient = T.grad(cost=cost, wrt=W)

W_updated = W - (gradient * 0.01)
updates = [(W, W_updated)]

f = th.function(inputs=[x, target], outputs=[y, cost, gradient], updates=updates, allow_input_downcast=True)

print("start weight:\n", W.get_value().todense())

for i in range(10):
    pred, c, g = f([[4.0, 2.0]], [[2.0, 4.0]])
    print("pred:", pred)
    print("cost:", c)
    print("grad:\n", g)

print("end weight:\n", W.get_value().todense())

我测试的例子很简单，矩阵值应该从0.5到1.0，其中成本将变为零。然而它发散，两个值都减小，梯度总是大约1.0和2.0。似乎梯度计算错误。如果我运行代码，我会得到以下输出：

start weight:
 [[ 0.   0.5]
 [ 0.5  0. ]]
pred: [[ 1.  2.]]
cost: 1.25
grad:
   (0, 1)   1.00000071339
  (1, 0)    2.00000143424
pred: [[ 0.97999999  1.91999994]]
cost: 1.3417000669408599
grad:
   (0, 1)   1.00000071343
  (1, 0)    2.00000143439
pred: [[ 0.95999997  1.83999989]]
cost: 1.4368001387634612
grad:
   (0, 1)   1.00000071347
  (1, 0)    2.00000143453
pred: [[ 0.93999996  1.75999983]]
cost: 1.5353002154685411
grad:
   (0, 1)   1.0000007135
  (1, 0)    2.00000143468
pred: [[ 0.91999994  1.67999977]]
cost: 1.637200297056838
grad:
   (0, 1)   1.00000071354
  (1, 0)    2.00000143483
pred: [[ 0.89999993  1.59999971]]
cost: 1.7425003835290889
grad:
   (0, 1)   1.00000071358
  (1, 0)    2.00000143498
pred: [[ 0.87999991  1.51999966]]
cost: 1.8512004748860316
grad:
   (0, 1)   1.00000071362
  (1, 0)    2.00000143513
pred: [[ 0.8599999  1.4399996]]
cost: 1.9633005711284035
grad:
   (0, 1)   1.00000071365
  (1, 0)    2.00000143528
pred: [[ 0.83999989  1.35999954]]
cost: 2.0788006722569428
grad:
   (0, 1)   1.00000071369
  (1, 0)    2.00000143543
pred: [[ 0.81999987  1.27999948]]
cost: 2.197700778272387
grad:
   (0, 1)   1.00000071373
  (1, 0)    2.00000143558
end weight:
 [[ 0.          0.39999993]
 [ 0.29999986  0.        ]]

Answer 1

我认为CSR和CSC稀疏矩阵在这种情况下应该起作用，因此sparse_matrix函数根据矩阵维度选择一个。但事实证明，如果我明确使用CSC并避免使用CSR，我的问题就解决了。然后，稀疏矩阵中的值收敛到1.0，成本消失。

具有稀疏权重矩阵发散的Theano神经网络

1 个答案: