Question

我在python中有以下代码

###############################################################################

"""
@INPUT:
    R     : a matrix to be factorized, dimension N x M
    P     : an initial matrix of dimension N x K
    Q     : an initial matrix of dimension M x K
    K     : the number of latent features
    steps : the maximum number of steps to perform the optimisation
    alpha : the learning rate
    beta  : the regularization parameter
@OUTPUT:
    the final matrices P and Q
"""
def matrix_factorization(R, P, Q, K, steps=5000, alpha=0.0002, beta=0.02):
    Q = Q.T
    for step in xrange(steps):
        for i in xrange(len(R)):
            for j in xrange(len(R[i])):
                if R[i][j] > 0:
                    eij = R[i][j] - numpy.dot(P[i,:],Q[:,j])
                    for k in xrange(K):
                        P[i][k] = P[i][k] + alpha * (2 * eij * Q[k][j] - beta * P[i][k])
                        Q[k][j] = Q[k][j] + alpha * (2 * eij * P[i][k] - beta * Q[k][j])
        eR = numpy.dot(P,Q)
        e = 0
        for i in xrange(len(R)):
            for j in xrange(len(R[i])):
                if R[i][j] > 0:
                    e = e + pow(R[i][j] - numpy.dot(P[i,:],Q[:,j]), 2)
                    for k in xrange(K):
                        e = e + (beta/2) * ( pow(P[i][k],2) + pow(Q[k][j],2) )
        if e < 0.001:
            break
    return P, Q.T

###############################################################################

此代码的链接如下： http://www.quuxlabs.com/blog/2010/09/matrix-factorization-a-simple-tutorial-and-implementation-in-python/

代码适用于小矩阵，但我有两个大矩阵P（15715 ，203）和Q（203,16384），当我尝试在P和Q上执行此代码时，它会给我以下错误

K=203

matrix_factorization(R, P, Q, K)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-00b8211f2507> in <module>()
----> 1 matrix_factorization(R, P, Q, K)

/Users/ajinkyachandrakantbobade/Desktop/random_choicefile/trial.py in matrix_factorization(R, P, Q, K, steps, alpha, beta)
     52             for j in xrange(len(R[i])):
     53                 if R[i][j] > 0:
---> 54                     eij = R[i][j] - numpy.dot(P[i,:],Q[:,j])
     55                     for k in xrange(K):
     56                         P[i][k] = P[i][k] + alpha * (2 * eij * Q[k][j] - beta * P[i][k])

/Users/ajinkyachandrakantbobade/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/frame.pyc in __getitem__(self, key)
   1967             return self._getitem_multilevel(key)
   1968         else:
-> 1969             return self._getitem_column(key)
   1970 
   1971     def _getitem_column(self, key):

/Users/ajinkyachandrakantbobade/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/frame.pyc in _getitem_column(self, key)
   1974         # get column
   1975         if self.columns.is_unique:
-> 1976             return self._get_item_cache(key)
   1977 
   1978         # duplicate columns & possible reduce dimensionality

/Users/ajinkyachandrakantbobade/Library/Enthought/Canopy_64bit/User/lib/python2.7/site-packages/pandas/core/generic.pyc in _get_item_cache(self, item)
   1087         """ return the cached item, item represents a label indexer """
   1088         cache = self._item_cache
-> 1089         res = cache.get(item)
   1090         if res is None:
   1091             values = self._data.get(item)

TypeError: unhashable type

有人可以帮忙解决这个错误吗？

Answer 1

您尝试乘法的矩阵的大小太大，而且您没有足够的内存来完成计算。一些可能有所帮助的事情：

获得更多内存
如果您的矩阵包含大量0，您可以尝试使用稀疏矩阵。它们是常规矩阵，仅存储具有不等于零的值的元素。 The documentation of scipy会为您提供相关信息。
看来你已经在使用python 64bit但是如果不是这样比python 32bit更好

Answer 2

您可以使用scipy包（例如scipy.sparse.coo_matrix（arg1 [，shape，dtype，copy]））将矩阵转换成稀疏矩阵这将允许在更大的数据集上使用MF，而不会遇到计算问题。

对于该方法的实现，我使用了Jason Feriante解释的非负矩阵分解 https://github.com/jferiante/Collaborative-Filtering-Machine-Learning/blob/master/python/nmf-learn.py

Python矩阵分解

2 个答案: