我可以从gensim运行LDA代码，并使用各自的关键字获得前10个主题。
现在我想进一步了解LDA算法的准确性，看看他们将哪些文档聚集到每个主题中。这在gensim LDA中是否可行？
基本上我想做类似的事情，但是在python中并使用gensim。
LDA with topicmodels, how can I see which topics different documents belong to?
答案 0 :(得分：26)
使用主题的概率，您可以尝试设置一些阈值并将其用作聚类基线，但我确信有比这种“hacky”方法更好的聚类方法。
from gensim import corpora, models, similarities
from itertools import chain
""" DEMO """
documents = ["Human machine interface for lab abc computer applications",
"A survey of user opinion of computer system response time",
"The EPS user interface management system",
"System and human system engineering testing of EPS",
"Relation of user perceived response time to error measurement",
"The generation of random binary unordered trees",
"The intersection graph of paths in trees",
"Graph minors IV Widths of trees and well quasi ordering",
"Graph minors A survey"]
# remove common words and tokenize
stoplist = set('for a of the and to in'.split())
texts = [[word for word in document.lower().split() if word not in stoplist]
for document in documents]
# remove words that appear only once
all_tokens = sum(texts, [])
tokens_once = set(word for word in set(all_tokens) if all_tokens.count(word) == 1)
texts = [[word for word in text if word not in tokens_once] for text in texts]
# Create Dictionary.
id2word = corpora.Dictionary(texts)
# Creates the Bag of Word corpus.
mm = [id2word.doc2bow(text) for text in texts]
# Trains the LDA models.
lda = models.ldamodel.LdaModel(corpus=mm, id2word=id2word, num_topics=3, \
update_every=1, chunksize=10000, passes=1)
# Prints the topics.
for top in lda.print_topics():
print top
print
# Assigns the topics to the documents in corpus
lda_corpus = lda[mm]
# Find the threshold, let's set the threshold to be 1/#clusters,
# To prove that the threshold is sane, we average the sum of all probabilities:
scores = list(chain(*[[score for topic_id,score in topic] \
for topic in [doc for doc in lda_corpus]]))
threshold = sum(scores)/len(scores)
print threshold
print
cluster1 = [j for i,j in zip(lda_corpus,documents) if i[0][1] > threshold]
cluster2 = [j for i,j in zip(lda_corpus,documents) if i[1][1] > threshold]
cluster3 = [j for i,j in zip(lda_corpus,documents) if i[2][1] > threshold]
print cluster1
print cluster2
print cluster3
[out]：
0.131*trees + 0.121*graph + 0.119*system + 0.115*user + 0.098*survey + 0.082*interface + 0.080*eps + 0.064*minors + 0.056*response + 0.056*computer
0.171*time + 0.171*user + 0.170*response + 0.082*survey + 0.080*computer + 0.079*system + 0.050*trees + 0.042*graph + 0.040*minors + 0.040*human
0.155*system + 0.150*human + 0.110*graph + 0.107*minors + 0.094*trees + 0.090*eps + 0.088*computer + 0.087*interface + 0.040*survey + 0.028*user
0.333333333333
['The EPS user interface management system', 'The generation of random binary unordered trees', 'The intersection graph of paths in trees', 'Graph minors A survey']
['A survey of user opinion of computer system response time', 'Relation of user perceived response time to error measurement']
['Human machine interface for lab abc computer applications', 'System and human system engineering testing of EPS', 'Graph minors IV Widths of trees and well quasi ordering']
只是为了更清楚：
# Find the threshold, let's set the threshold to be 1/#clusters,
# To prove that the threshold is sane, we average the sum of all probabilities:
scores = []
for doc in lda_corpus
for topic in doc:
for topic_id, score in topic:
scores.append(score)
threshold = sum(scores)/len(scores)
以上代码是所有单词的分数和所有文档的所有主题的总和。 然后将总和标准化为分数。
答案 1 :(得分：8)
如果你想使用
的伎俩cluster1 = [j for i,j in zip(lda_corpus,documents) if i[0][1] > threshold]
cluster2 = [j for i,j in zip(lda_corpus,documents) if i[1][1] > threshold]
cluster3 = [j for i,j in zip(lda_corpus,documents) if i[2][1] > threshold]
在alvas的上一个答案中，请确保在LdaModel中设置minimum_probability = 0
gensim.models.ldamodel.LdaModel(corpus,
num_topics=num_topics, id2word = dictionary,
passes=2, minimum_probability=0)
否则lda_corpus和文档的维度可能不一致，因为gensim会抑制概率低于minimum_probability的任何语料库。
将文档分组到主题中的另一种方法是根据最大概率分配主题
lda_corpus = [max(prob,key=lambda y:y[1])
for prob in lda[mm] ]
playlists = [[] for i in xrange(topic_num])]
for i, x in enumerate(lda_corpus):
playlists[x[0]].append(documents[i])
注意lda[mm]粗略地说是列表或2D矩阵。行数是文档数，列数是主题数。例如，每个矩阵元素是(3,0.82)形式的元组。这里3指的是主题索引，0.82指的是该主题的相应概率。默认情况下，minimum_probability=0.01中省略了lda[mm]和概率小于0.01的任何元组。如果您使用具有最大概率的分组方法，则可以将其设置为1 /＃主题。
答案 2 :(得分：0)
lda_corpus [i] [j]的形式为[（0，t1），（0，t2）...，（0，t10），....（n，t10）]，其中第一项表示文档索引，第二个术语表示该主题在该特定文档中的概率。