如何对语料库进行词法化?

时间:2019-04-17 18:10:06

标签: python-3.x nltk tokenize corpus

我需要对一个语料库(特别是布朗语料库)进行词缀化。怎样进行?

我使用了brown.paras()方法来获取段落列表并将其存储在变量中。

list1 = brown.paras()

这是我对list1变量的输出

[[['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday', 'an', 'investigation', 'of', "Atlanta's", 'recent', 'primary', 'election', 'produced', '``', 'no', 'evidence', "''", 'that', 'any', 'irregularities', 'took', 'place', '.']], [['The', 'jury', 'further', 'said', 'in', 'term-end', 'presentments', 'that', 'the', 'City', 'Executive', 'Committee', ',', 'which', 'had', 'over-all', 'charge', 'of', 'the', 'election', ',', '``', 'deserves', 'the', 'praise', 'and', 'thanks', 'of', 'the', 'City', 'of', 'Atlanta', "''", 'for', 'the', 'manner', 'in', 'which', 'the', 'election', 'was', 'conducted', '.']], ...]

我需要对此标记化,小写标记并对其进行词形化

0 个答案:

没有答案