我试图使用scikit-learn的DBSCAN实现来集群化一堆文档。首先，我使用scikit-learn的TfidfVectorizer创建TF-IDF矩阵（它是numpy.float64类型的163405x13029稀疏矩阵）。然后我尝试聚类该矩阵的特定子集。当子集较小时（例如，最多几千行），事情就可以正常工作。但是对于大型子集（有数万行），我得到ValueError: could not convert integer scalar。
这里是完整的追溯（idxs是一个索引列表）：
ValueError Traceback (most recent call last)
<ipython-input-1-73ee366d8de5> in <module>()
193 # use descriptions to clusterize items
194 ncm_clusterizer = DBSCAN()
--> 195 ncm_clusterizer.fit_predict(tfidf[idxs])
196 idxs_clusters = list(zip(idxs, ncm_clusterizer.labels_))
197 for e in idxs_clusters:
/usr/local/lib/python3.4/site-packages/sklearn/cluster/dbscan_.py in fit_predict(self, X, y, sample_weight)
294 cluster labels
295 """
--> 296 self.fit(X, sample_weight=sample_weight)
297 return self.labels_
/usr/local/lib/python3.4/site-packages/sklearn/cluster/dbscan_.py in fit(self, X, y, sample_weight)
264 X = check_array(X, accept_sparse='csr')
265 clust = dbscan(X, sample_weight=sample_weight,
--> 266 **self.get_params())
267 self.core_sample_indices_, self.labels_ = clust
268 if len(self.core_sample_indices_):
/usr/local/lib/python3.4/site-packages/sklearn/cluster/dbscan_.py in dbscan(X, eps, min_samples, metric, algorithm, leaf_size, p, sample_weight, n_jobs)
136 # This has worst case O(n^2) memory complexity
137 neighborhoods = neighbors_model.radius_neighbors(X, eps,
--> 138 return_distance=False)
139
140 if sample_weight is None:
/usr/local/lib/python3.4/site-packages/sklearn/neighbors/base.py in radius_neighbors(self, X, radius, return_distance)
584 if self.effective_metric_ == 'euclidean':
585 dist = pairwise_distances(X, self._fit_X, 'euclidean',
--> 586 n_jobs=self.n_jobs, squared=True)
587 radius *= radius
588 else:
/usr/local/lib/python3.4/site-packages/sklearn/metrics/pairwise.py in pairwise_distances(X, Y, metric, n_jobs, **kwds)
1238 func = partial(distance.cdist, metric=metric, **kwds)
1239
-> 1240 return _parallel_pairwise(X, Y, func, n_jobs, **kwds)
1241
1242
/usr/local/lib/python3.4/site-packages/sklearn/metrics/pairwise.py in _parallel_pairwise(X, Y, func, n_jobs, **kwds)
1081 if n_jobs == 1:
1082 # Special case to avoid picklability checks in delayed
-> 1083 return func(X, Y, **kwds)
1084
1085 # TODO: in some cases, backend='threading' may be appropriate
/usr/local/lib/python3.4/site-packages/sklearn/metrics/pairwise.py in euclidean_distances(X, Y, Y_norm_squared, squared, X_norm_squared)
243 YY = row_norms(Y, squared=True)[np.newaxis, :]
244
--> 245 distances = safe_sparse_dot(X, Y.T, dense_output=True)
246 distances *= -2
247 distances += XX
/usr/local/lib/python3.4/site-packages/sklearn/utils/extmath.py in safe_sparse_dot(a, b, dense_output)
184 ret = a * b
185 if dense_output and hasattr(ret, "toarray"):
--> 186 ret = ret.toarray()
187 return ret
188 else:
/usr/local/lib/python3.4/site-packages/scipy/sparse/compressed.py in toarray(self, order, out)
918 def toarray(self, order=None, out=None):
919 """See the docstring for `spmatrix.toarray`."""
--> 920 return self.tocoo(copy=False).toarray(order=order, out=out)
921
922 ##############################################################
/usr/local/lib/python3.4/site-packages/scipy/sparse/coo.py in toarray(self, order, out)
256 M,N = self.shape
257 coo_todense(M, N, self.nnz, self.row, self.col, self.data,
--> 258 B.ravel('A'), fortran)
259 return B
260
ValueError: could not convert integer scalar
我使用的是Python 3.4.3（在Red Hat上），scipy 0.18.1，scikit-learn 0.18.1。
我尝试了建议使用here的猴子补丁但是没有用。
在Google上搜索我发现bugfix显然为其他类型的稀疏矩阵（如csr）解决了同样的问题，但不适用于咕咕声。
我已尝试按照建议的here向DBSCAN提供稀疏半径邻域图（而不是特征矩阵），但发生了相同的错误。
我已经尝试了HDBSCAN，但同样的错误发生了。
我该如何解决这个问题或绕过它？
答案 0 :(得分：3)
即使实现允许，DBSCAN也可能会在如此高维度的数据上产生不良结果（从统计的角度来看，因为维数的诅咒）。
相反，我会建议您使用TruncatedSVD类将TF-IDF特征向量的维数降低到50或100个组件，然后对结果应用DBSCAN。