在应用ngram

Question

目前我正在从excel文件中读取文本并将bigram应用于它。 finalList 具有以下示例代码中使用的列表，其中包含从输入excel文件中读取的输入词列表。

借助以下库删除输入中的停用词：

from nltk.corpus import stopwords

bigram逻辑应用于单词输入文本列表

bigram=ngrams(finalList ,2)

输入文字：我完成了我的端到端流程。

当前输出：完成结束，结束，结束过程。

期望的输出：完成端到端，端到端流程。

这意味着一些像（端到端）这样的词组应被视为一个词。

Answer 1

要解决您的问题，您必须使用正则表达式清理停用词。见这个例子：

 import re
 text = 'I completed my end-to-end process..:?' 
 pattern = re.compile(r"\.*:\?*") # to remove zero or more instances of such stop words, the hyphen is not included in the stop words. 
 new_text = re.sub(pattern, '', text)
 print(new_text)
 'I completed my end-to-end process'


 # Now you can generate bigrams manually.
 # 1. Tokanize the new text
 tok = new_text.split()
 print(tok) # If the size of token is huge, just print the first five ones, like this print(tok[:5])
 ['I', 'completed', 'my', 'end-to-end', 'process']

 # 2. Loop over the list and generate bigrams, store them in a var called bigrams
 bigrams = []
 for i in range(len(tok) - 1):  # -1 to avoid index error
     bigram = tok[i] + ' ' + tok[i + 1]  
     bigrams.append(bigram)


 # 3. Print your bigrams
 for bi in bigrams:
     print(bi, end = ', ')

I completed, completed my, my end-to-end, end-to-end process,

我希望这有帮助！