Question

我正在使用这两个函数来查找相似的单词并返回不同的列表。我想知道这些函数是按大多数到最不频繁的关联排序的吗？

Answer 1

ContextIndex.similar_words(word)计算每个单词的相似度得分，作为每个上下文中频率乘积的总和。 Text.similar()只计算单词共享的唯一上下文的数量。

similar_words()似乎包含NLTK 2.0中的错误。请参阅nltk/text.py中的定义：

def similar_words(self, word, n=20):
    scores = defaultdict(int)
    for c in self._word_to_contexts[self._key(word)]:
        for w in self._context_to_words[c]:
            if w != word:
                print w, c, self._context_to_words[c][word], self._context_to_words[c][w]
                scores[w] += self._context_to_words[c][word] * self._context_to_words[c][w]
    return sorted(scores, key=scores.get)[:n]

返回的单词列表应按相似度分数的降序排序。将return语句替换为：

return sorted(scores, key=scores.get)[::-1][:n]

在similar()中，对similar_words()的调用已被注释掉，可能是由于此错误。

def similar(self, word, num=20):
    if '_word_context_index' not in self.__dict__:
        print 'Building word-context index...'
        self._word_context_index = ContextIndex(self.tokens,
                                                filter=lambda x:x.isalpha(),
                                                key=lambda s:s.lower())

#   words = self._word_context_index.similar_words(word, num)

    word = word.lower()
    wci = self._word_context_index._word_to_contexts
    if word in wci.conditions():
        contexts = set(wci[word])
        fd = FreqDist(w for w in wci.conditions() for c in wci[w]
                      if c in contexts and not w == word)
        words = fd.keys()[:num]
        print tokenwrap(words)
    else:
        print "No matches"

注意：在FreqDist中，与dict不同，keys()会返回已排序的列表。

示例：

import nltk text = nltk.Text(word.lower() for word in nltk.corpus.brown.words()) text.similar('woman') similar_words = text._word_context_index.similar_words('woman') print ' '.join(similar_words)

<强>输出：

man day time year car moment world family house boy child country job state girl place war way case question # Text.similar() #man ('a', 'who') 9 39 # output from similar_words(); see following explanation #girl ('a', 'who') 9 6 #[...] man number time world fact end year state house way day use part kind boy matter problem result girl group # ContextIndex.similar_words()

fd，similar()中的频率分布，是每个单词的上下文数量的总和：

fd = [('man', 52), ('day', 30), ('time', 30), ('year', 28), ('car', 24), ('moment', 24), ('world', 23) ...]

对于每个上下文中的每个单词，similar_words()计算频率乘积之和：

man ('a', 'who') 9 39 # 'a man who' occurs 39 times in text; # 'a woman who' occurs 9 times # Similarity score for the context is the product: # score['man'] = 9 * 39 girl ('a', 'who') 9 6 writer ('a', 'who') 9 4 boy ('a', 'who') 9 3 child ('a', 'who') 9 2 dealer ('a', 'who') 9 2 ... man ('a', 'and') 6 11 # score += 6 * 11 ... man ('a', 'he') 4 6 # score += 4 * 6 ... [49 more occurrences of 'man']

从频率中排序的NLTK中的Text.similar（）和ContextIndex.similar_words（）生成的单词？

1 个答案: