Question

我有2本txt格式的书（6000多行）。我想使用Python将每个单词与其相关性（使用td idf算法）联系起来，并按降序排序。我试过这段代码

#- * -coding: utf - 8 - * -
    from __future__
import division, unicode_literals
import math
from textblob
import TextBlob as tb

def tf(word, blob):
    return blob.words.count(word) / len(blob.words)

def n_containing(word, bloblist):
    return sum(1
        for blob in bloblist
        if word in blob)

def idf(word, bloblist):
    return math.log(len(bloblist) / (1 + n_containing(word, bloblist)))

def tfidf(word, blob, bloblist):
    return tf(word, blob) * idf(word, bloblist)

document1 = tb(""
    "FULL BOOK1 TEST"
    "")
document2 = tb(""
    "FULL BOOK2 TEST"
    "")



bloblist = [document1, document2]
for i, blob in enumerate(bloblist):
    with open("result.txt", 'w') as textfile:
    print("Top words in document {}".format(i + 1))
scores = {
    word: tfidf(word, blob, bloblist) for word in blob.words
}
sorted_words = sorted(scores.items(), key = lambda x: x[1], reverse = True)
for word, score in sorted_words:
    textfile.write("Word: {}, TF-IDF: {}".format(word, round(score, 5)) + "\n")

我在这里https://stevenloria.com/tf-idf/发现了一些变化，但需要花费很多时间，几分钟后，它会崩溃说TypeError: coercing to Unicode: need string or buffer, float found。为什么呢？

我也尝试通过python https://github.com/mccurdyc/tf-idf/调用这个Java程序。该程序有效，但输出不正确：有很多单词应该具有较高的相关性级别，而是分为0相关性。

有没有办法修复Python代码？或者，你能否建议我另一个正确执行我想要的tf-idf实现？

Tf Idf在一组文件中查找单词相关性

0 个答案: