Question

我有一个相当大的数据集（1841000 * 32矩阵），我希望对其运行分层聚类算法。 sklearn.cluster中的AgglomerativeClustering类和FeatureAgglomeration类都给出以下错误。

    ---------------------------------------------------------------------------
    MemoryError                               Traceback (most recent call last)
    <ipython-input-10-85ab7b694cf1> in <module>()
  1 
  2 
    ----> 3 mat_red = manifold.SpectralEmbedding(n_components=2).fit_transform(mat)
  4 clustering.fit(mat_red,y = None)

 ~/anaconda3/lib/python3.6/site-packages/sklearn/manifold/spectral_embedding_.py in fit_transform(self, X, y)
525         X_new : array-like, shape (n_samples, n_components)
526         """
--> 527         self.fit(X)
528         return self.embedding_

  ~/anaconda3/lib/python3.6/site-packages/sklearn/manifold/spectral_embedding_.py in fit(self, X, y)
498                               "name or a callable. Got: %s") % self.affinity)
499 
--> 500         affinity_matrix = self._get_affinity_matrix(X)
501         self.embedding_ = spectral_embedding(affinity_matrix,
502   n_components=self.n_components,

~/anaconda3/lib/python3.6/site-packages/sklearn/manifold/spectral_embedding_.py in _get_affinity_matrix(self, X, Y)
450     self.affinity_matrix_ = kneighbors_graph(X, self.n_neighbors_,
451                                                          include_self=True,
--> 452                                                          n_jobs=self.n_jobs)
453                 # currently only symmetric affinity_matrix supported
454                 self.affinity_matrix_ = 0.5 * (self.affinity_matrix_ +

~/anaconda3/lib/python3.6/site-packages/sklearn/neighbors/graph.py in kneighbors_graph(X, n_neighbors, mode, metric, p, metric_params, include_self, n_jobs)
101 
102     query = _query_include_self(X, include_self)
--> 103     return X.kneighbors_graph(X=query, n_neighbors=n_neighbors, mode=mode)
104 
105 

~/anaconda3/lib/python3.6/site-packages/sklearn/neighbors/base.py in kneighbors_graph(self, X, n_neighbors, mode)
482         # construct CSR matrix representation of the k-NN graph
483         if mode == 'connectivity':
--> 484             A_data = np.ones(n_samples1 * n_neighbors)
485             A_ind = self.kneighbors(X, n_neighbors, return_distance=False)
486 

   ~/anaconda3/lib/python3.6/site-packages/numpy/core/numeric.py in ones(shape, dtype, order)
186 
187     """
    --> 188     a = empty(shape, dtype, order)
189     multiarray.copyto(a, 1, casting='unsafe')
190     return a

MemoryError:

我的RAM为8GB，在64GB系统上运行该错误。我意识到层次聚类在计算上很昂贵，不建议用于大型数据集，但是我需要一次创建所有数据的树状图。我正在使用ORB功能从一堆视觉单词中创建词汇树。如果还有其他方法可以解决此问题或解决错误，请说明！谢谢。

Answer 1

我在运行聚集集群时遇到了类似的问题。我的解决方案是使用train_test_split在数据的一小部分上运行聚类算法，然后使用KNN将标签从AC扩展到其余数据。可以正常工作，不确定所使用的数据是否适合该处理方式。我的扩展代码是：

X_train, X_test, y_train, y_test = \
    train_test_split(X, y, 
                     test_size=test_size, random_state=42) 
AC = AgglomerativeClustering(n_clusters=n_clusters, linkage='ward')
AC.fit(X_train)
labels = AC.labels_

KN = KNeighborsClassifier(n_neighbors=n_neighbors)
KN.fit(X_train,labels)
labels2 = KN.predict(X)

分层群集Python 3.6期间发生内存错误

1 个答案: