将稀疏矩阵转换为密集矩阵时的MemoryError? (numpy,scikit)

时间:2013-10-09 19:52:09

标签: python numpy scikit-learn

lr = lm.LogisticRegression(penalty='l2', dual=True, tol=0.0001, 
                             C=1, fit_intercept=True, intercept_scaling=1.0, 
                             class_weight=None, random_state=None)

rd = AdaBoostClassifier( base_estimator=lr, 
                                           learning_rate=1, 
                                           n_estimators=20, 
                                           algorithm="SAMME")
##here, i am deleting unnecesseary objects
##print X.shape
##(7395, 412605)
print "20 Fold CV Score: ", np.mean(cross_validation.cross_val_score(rd, X, y, cv=20, scoring='roc_auc'))

当我运行时,我收到此错误:

  

TypeError:传递了稀疏矩阵,但需要密集数据。使用   X.toarray()转换为密集的numpy数组。

然后,我改变了我的代码:

print "20 Fold CV Score: ", np.mean(cross_validation.cross_val_score(rd, X.toarray(), y, cv=20, scoring='roc_auc'))

现在,我有以下例外:

  File "/usr/lib/python2.7/dist-packages/scipy/sparse/compressed.py", line 559, in toarray
    return self.tocoo(copy=False).toarray(order=order, out=out)
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/coo.py", line 235, in toarray
    B = self._process_toarray_args(order, out)
  File "/usr/lib/python2.7/dist-packages/scipy/sparse/base.py", line 628, in _process_toarray_args
    return np.zeros(self.shape, dtype=self.dtype, order=order)
MemoryError

有任何解决问题的建议吗?

1 个答案:

答案 0 :(得分:11)

MemoryError表示系统上没有足够的RAM来分配矩阵。为什么?好吧,7395 x 412605矩阵有3,051,213,975个元素。如果它们处于默认的float64(通常是C中的double)数据类型,则为22.7GB。如果转换为精度较低的float32 s(通常是C中的float),则为11.4GB;也许这可以在你的机器上操作。但它仍然会很慢。

似乎AdaBoostClassifier不支持稀疏输入(如您所见in the code here)。我不知道算法是否需要密集表示,或者只是实现假设的那样。