我有JSON格式的亚马逊用户评论数据,我将其导入到pandas数据框并使用它来训练文本分类模型。我正在尝试在使用该数据训练模型之前预处理用户评论文本。我在这里有两个问题:
1)我已经编写了一个代码来使用Python中的Textblob库来检测它的语言,它运行良好但耗费了大量时间。请告诉我是否有最佳方法。我在python中使用Textblob库,代码是:
from textblob import TextBlob
def detect_language(text):
if len(text)>3:
r=TextBlob(text)
lang = r.detect_language()
return lang
dataset['language']=dataset.reviewText.apply(lambda x: detect_language(x))
2)我想在训练模型之前将我的单词弄清楚。但是,如果我们使用带有单词的词性标注,NLTK中的词形还原可以正常工作,我会尝试如下但是会出现一些错误:
from nltk import pos_tag
from nltk.stem import WordNetLemmatizer
text='my name is shubham'
text=pos_tag(text.split())
wl=WordNetLemmatizer()
for i in text:
print(wl.lemmatize(i))
我在这里被标记为:
[('my', 'PRP$'), ('name', 'NN'), ('is', 'VBZ'), ('shubham', 'JJ')]
并且在进行词形还原时我会收到错误:
AttributeError: 'tuple' object has no attribute 'endswith'
请您建议一种有效的词形还原方法。 以下是我正在执行语言检测和词形还原的示例数据:
overall reviewText
5 Not much to write about here, but it does exac...
5 The product does exactly as it should and is q...
5 The primary job of this device is to block the...
5 Nice windscreen protects my MXL mic and preven...
5 This pop filter is great. It looks and perform...
答案 0 :(得分:1)
from nltk import pos_tag, word_tokenize
from nltk.stem import WordNetLemmatizer
wnl = WordNetLemmatizer()
def penn2morphy(penntag):
""" Converts Penn Treebank tags to WordNet. """
morphy_tag = {'NN':'n', 'JJ':'a',
'VB':'v', 'RB':'r'}
try:
return morphy_tag[penntag[:2]]
except:
return 'n'
def lemmatize_sent(text):
# Text input is string, returns lowercased strings.
return [wnl.lemmatize(word.lower(), pos=penn2morphy(tag))
for word, tag in pos_tag(word_tokenize(text))]
将字符串的数据帧列变为lemmatize。
df['lemmas'] = df['text'].apply(lemmatize_sent)