Question

我试图显示here显示的Q6配方，但我的语料库仍然以[]的形式返回，即使我已经检查过它似乎确实正在正确阅读文档。

所以我的代码是：

def iter_documents(top_directory):
    """Iterate over all documents, yielding a document (=list of utf8 tokens) at a time."""
    for root, dirs, files in os.walk(top_directory):
        for file in filter(lambda file: file.endswith('.txt'), files):
            document = open(os.path.join(root, file)).read() # read the entire document, as one big string
            yield utils.tokenize(document, lower=True) # or whatever tokenization suits you
class MyCorpus(object):
    # Used to create the object
    def __init__(self, top_dir):
        self.top_dir = top_dir
        self.dictionary = corpora.Dictionary(iter_documents(top_dir))
        self.dictionary.filter_extremes(no_below=1, keep_n=30000) # check API docs for pruning params
# Used if you ever need to iterate through the values
def __iter__(self):
    for tokens in iter_documents(self.top_dir):
        yield self.dictionary.doc2bow(tokens)

我用来测试的文本文件是this。

Answer 1

好的我明白了。将第12行更改为：self.dictionary.filter_extremes(no_below=0, no_above=1,keep_n=30000)

因为我只有1个文件，所以它被过滤掉了。见this

Gensim - 迭代多个文档

1 个答案: