我正在使用以下代码来获取用户帖子的排序列表。
model = doc2vec.Doc2Vec.load(doc2vec_model_name)
doc_vectors = model.docvecs.doctag_syn0
doc_tags = model.docvecs.offset2doctag
for w, sim in model.docvecs.most_similar(positive=[model.infer_vector('phone_comments')], topn=4000):
print(w, sim)
fw.write(w)
fw.write(" (")
fw.write(str(sim))
fw.write(")")
fw.write("\n")
fw.close()
但是,我也在列表的第6位获得了向量"phone comments"
(我用来查找最近的邻居)。我在代码中有任何错误吗?还是Gensim中的一个问题(因为向量不能是其自身的邻居)?
编辑
Doc2vec模型训练代码
######Preprocessing
docs = []
analyzedDocument = namedtuple('AnalyzedDocument', 'words tags')
for key, value in my_d.items():
value = re.sub("[^1-9a-zA-Z]"," ", value)
words = value.lower().split()
tags = key.replace(' ', '_')
docs.append(analyzedDocument(words, tags.split(' ')))
sentences = [] # Initialize an empty list of sentences
######Get n-grams
#Get list of lists of tokenised words. 1 sentence = 1 list
for item in docs:
sentences.append(item.words)
#identify bigrams and trigrams (trigram_sentences_project)
trigram_sentences_project = []
bigram = Phrases(sentences, min_count=5, delimiter=b' ')
trigram = Phrases(bigram[sentences], min_count=5, delimiter=b' ')
for sent in sentences:
bigrams_ = bigram[sent]
trigrams_ = trigram[bigram[sent]]
trigram_sentences_project.append(trigrams_)
paper_count = 0
for item in trigram_sentences_project:
docs[paper_count] = docs[paper_count]._replace(words=item)
paper_count = paper_count+1
# Train model
model = doc2vec.Doc2Vec(docs, size = 100, window = 300, min_count = 5, workers = 4, iter = 20)
#Save the trained model for later use to take the similarity values
model_name = user_defined_doc2vec_model_name
model.save(model_name)
答案 0 :(得分:1)
infer_vector()
方法需要一个令牌列表,就像用于训练模型的文本示例(通常为words
对象)的TaggedDocument
属性一样。
您正在提供一个简单的字符串'phone_comments'
,看起来像列表infer_vector()
一样,在['p', 'h', 'o', 'n', 'e', '_', 'c', 'o', 'm', 'm', 'e', 'n', 't', 's']
上也是如此。因此,most_similar()
的原始向量可能是垃圾。
此外,您没有获取输入'phone_comments'
,而是获取了不同的字符串'phone comments'
。如果这是模型中的标记名,则必须在模型训练期间提供tag
。它与phone_comments
的表面相似性可能毫无意义-它们是不同的字符串。
(但是这也可能暗示您的培训也有问题,并且将本应由words=['phone', 'comments']
改成words=['p', 'h', 'o', 'n', 'e', ' ', 'c', 'o', 'm', 'm', 'e', 'n', 't', 's']
的文本进行了培训。)