Question

我正在使用Windows 10 64位12gb RAM内核i5。

现在对大约30k的亚马逊数据集进行即时测试

训练数据中的246621项，测试数据中的61656项

我在scikit中尝试了其他机器学习，学习效果很好，但在Knn遇到内存错误的问题。

我的代码

knn = KNeighborsClassifier(n_neighbors=5).fit(X_train_tfidf, y_train)
prediction['knn'] = knn.predict(X_test_tfidf)
accuracy_score(y_test, prediction['knn'])*100

我的错误

MemoryError                               Traceback (most recent call last)
<ipython-input-13-4d958e7f8f5b> in <module>()
      1 knn = KNeighborsClassifier(n_neighbors=5).fit(X_train_tfidf, y_train)
----> 2 prediction['knn'] = knn.predict(X_test_tfidf)
      3 accuracy_score(y_test, prediction['knn'])*100

~\Anaconda3\lib\site-packages\sklearn\neighbors\classification.py in predict(self, X)
    143         X = check_array(X, accept_sparse='csr')
    144 
--> 145         neigh_dist, neigh_ind = self.kneighbors(X)
    146 
    147         classes_ = self.classes_

~\Anaconda3\lib\site-packages\sklearn\neighbors\base.py in kneighbors(self, X, n_neighbors, return_distance)
    355             if self.effective_metric_ == 'euclidean':
    356                 dist = pairwise_distances(X, self._fit_X, 'euclidean',
--> 357                                           n_jobs=n_jobs, squared=True)
    358             else:
    359                 dist = pairwise_distances(

~\Anaconda3\lib\site-packages\sklearn\metrics\pairwise.py in pairwise_distances(X, Y, metric, n_jobs, **kwds)
   1245         func = partial(distance.cdist, metric=metric, **kwds)
   1246 
-> 1247     return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
   1248 
   1249 

~\Anaconda3\lib\site-packages\sklearn\metrics\pairwise.py in _parallel_pairwise(X, Y, func, n_jobs, **kwds)
   1088     if n_jobs == 1:
   1089         # Special case to avoid picklability checks in delayed
-> 1090         return func(X, Y, **kwds)
   1091 
   1092     # TODO: in some cases, backend='threading' may be appropriate

~\Anaconda3\lib\site-packages\sklearn\metrics\pairwise.py in euclidean_distances(X, Y, Y_norm_squared, squared, X_norm_squared)
    244         YY = row_norms(Y, squared=True)[np.newaxis, :]
    245 
--> 246     distances = safe_sparse_dot(X, Y.T, dense_output=True)
    247     distances *= -2
    248     distances += XX

~\Anaconda3\lib\site-packages\sklearn\utils\extmath.py in safe_sparse_dot(a, b, dense_output)
    133     """
    134     if issparse(a) or issparse(b):
--> 135         ret = a * b
    136         if dense_output and hasattr(ret, "toarray"):
    137             ret = ret.toarray()

~\Anaconda3\lib\site-packages\scipy\sparse\base.py in __mul__(self, other)
    367             if self.shape[1] != other.shape[0]:
    368                 raise ValueError('dimension mismatch')
--> 369             return self._mul_sparse_matrix(other)
    370 
    371         # If it's a list or whatever, treat it like a matrix

~\Anaconda3\lib\site-packages\scipy\sparse\compressed.py in _mul_sparse_matrix(self, other)
    538                                     maxval=nnz)
    539         indptr = np.asarray(indptr, dtype=idx_dtype)
--> 540         indices = np.empty(nnz, dtype=idx_dtype)
    541         data = np.empty(nnz, dtype=upcast(self.dtype, other.dtype))
    542 

MemoryError:

Answer 1

您可以尝试增加在KNeighborsClassifier docs上建议的leaf_size

leaf_size：int，可选（默认= 30）
Leaf size passed to BallTree or KDTree. This can affect the speed of the construction and query, as well as the memory required to store
那棵树。最佳值取决于问题的性质。

首先设置algorithm = "kd_tree"，然后尝试例如leaf_size = 300

内存错误：sklearn Knearest邻居knn

1 个答案: