训练t-sne模型时的MemoryError

时间:2017-08-08 12:41:14

标签: python memory error-handling jupyter-notebook

这是我的Jupyter代码,它触发了MemoryError

all_word_vectors_matrix_2d = tsne.fit_transform(all_word_vectors_matrix)

这是错误

MemoryError                               Traceback (most recent call last)
<ipython-input-39-c9bac182acc1> in <module>()
----> 1 all_word_vectors_matrix_2d = 
tsne.fit_transform(all_word_vectors_matrix)

C:\Users\Shilpa Suresh\Anaconda3\lib\site-packages\sklearn\manifold\t_sne.py 
in fit_transform(self, X, y)
    882             Embedding of the training data in low-dimensional space.
    883         """
--> 884         embedding = self._fit(X)
    885         self.embedding_ = embedding
    886         return self.embedding_

 C:\Users\Shilpa Suresh\Anaconda3\lib\site-
packages\sklearn\manifold\t_sne.py in _fit(self, X, skip_num_points)
    764                 neighbors_nn = neighbors_nn[:, 1:]
    765             P = _joint_probabilities_nn(distances, neighbors_nn,
--> 766                                         self.perplexity, self.verbose)
    767         else:
    768             P = _joint_probabilities(distances, self.perplexity, self.verbose)

C:\Users\Shilpa Suresh\Anaconda3\lib\site-packages\sklearn\manifold\t_sne.py 
in _joint_probabilities_nn(distances, neighbors, desired_perplexity, verbose)
     93     neighbors = astype(neighbors, np.int64, copy=False)
     94     conditional_P = _utils._binary_search_perplexity(
---> 95         distances, neighbors, desired_perplexity, verbose)
     96     m = "All probabilities should be finite"
     97     assert np.all(np.isfinite(conditional_P)), m

sklearn\manifold\_utils.pyx in 
sklearn.manifold._utils._binary_search_perplexity 
(sklearn\manifold\_utils.c:2683)()

sklearn\manifold\_utils.pyx in 
sklearn.manifold._utils._binary_search_perplexity 
(sklearn\manifold\_utils.c:1697)()

MemoryError: 

如何修复此内存错误,这是什么意思?如果您需要更多代码来理解错误,请随时告诉我

更新:我刚刚编辑了我的代码,将all_word_vectors_matrix拆分为两个,然后在all_word_vectors_matrix的前半部分运行了tsne.fit_transform()方法,但这似乎无法解决问题。

all_word_vectors_matrix_list = np.split(all_word_vectors_matrix, indices_or_sections=2)
all_word_vectors_matrix_2d = tsne.fit_transform(all_word_vectors_matrix_list[1])

0 个答案:

没有答案