Question

我在主题建模和Gensim方面还很陌生。因此，我仍在尝试理解许多概念。我试图在我的语料库上运行gensim的LDA模型，该模型包含约25,446,114条推文。我使用gensim创建了一个流式语料库和id2word字典。我正在使用num_topics = 100，块大小= 85000（一次加载85000条tweets）

我正在使用 Gensim：3.5.0 脾气暴躁：1.15.3

这是语料库和id2word词典的链接：https://drive.google.com/drive/folders/1FrJ8gJbiDqp3VC5syOjRVcQPcESdYOYa?usp=sharing

我不知道我在做什么错或如何解决这个问题。主题diff首先到达inf，然后到nan，我开始得到相同的主题。请帮忙!!

这是代码：

import pprint
import logging
import gensim
logging.basicConfig(filename='gensim.log',
                    format="%(asctime)s:%(levelname)s:%(message)s",
                    level=logging.INFO)
corpus = gensim.corpora.MmCorpus('disasterTweets.mm')
id2word = gensim.corpora.Dictionary.load('disasterTweets.dict')
id2word.filter_tokens(bad_ids=[id2word.token2id['eofeofeof']])
print('eofeofeof' in id2word.token2id)

lda_model = gensim.models.LdaMulticore(corpus=corpus,
                                       id2word=id2word,
                                       chunksize=85000,
                                       num_topics=100)
pprint.pprint(lda_model.print_topics())

这是我收到的错误：

/home/ec2-user/env/lib/python3.7/site-packages/gensim/models/ldamodel.py:1023: RuntimeWarning: divide by zero encountered in log
 diff = np.log(self.expElogbeta)
/home/ec2-user/env/lib/python3.7/site-packages/gensim/models/ldamodel.py:690: RuntimeWarning: overflow encountered in add
  sstats[:, ids] += np.outer(expElogthetad.T, cts / phinorm)
/home/ec2-user/env/lib/python3.7/site-packages/gensim/models/ldamodel.py:700: RuntimeWarning: invalid value encountered in multiply
  sstats *= self.expElogbeta
/home/ec2-user/env/lib/python3.7/site-packages/gensim/models/ldamodel.py:690: RuntimeWarning: overflow encountered in add
  sstats[:, ids] += np.outer(expElogthetad.T, cts / phinorm)
/home/ec2-user/env/lib/python3.7/site-packages/gensim/models/ldamodel.py:700: RuntimeWarning: invalid value encountered in multiply
  sstats *= self.expElogbeta
Process ForkPoolWorker-30:
Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/Cellar/python/3.7.0/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/linuxbrew/.linuxbrew/Cellar/python/3.7.0/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/home/linuxbrew/.linuxbrew/Cellar/python/3.7.0/lib/python3.7/multiprocessing/pool.py", line 105, in worker
    initializer(*initargs)
  File "/home/ec2-user/env/lib/python3.7/site-packages/gensim/models/ldamulticore.py", line 333, in worker_e_step
    worker_lda.do_estep(chunk)  # TODO: auto-tune alpha?
  File "/home/ec2-user/env/lib/python3.7/site-packages/gensim/models/ldamodel.py", line 725, in do_estep
    gamma, sstats = self.inference(chunk, collect_sstats=True)
  File "/home/ec2-user/env/lib/python3.7/site-packages/gensim/models/ldamodel.py", line 662, in inference
    expElogbetad = self.expElogbeta[:, ids]
IndexError: index 287500 is out of bounds for axis 1 with size 287500

Answer 1

从我对阅读Gensim Github问题页面issue 217中的线程的理解来看，这似乎是一个错误，那里的一些人报告说通过更改某些参数已解决了该问题。请先检查一下，看看那里的建议是否可以解决您的问题。

Gensim LDA模型主题差异导致nan

1 个答案: