我正在创建一类Unigram Tagger。我相信我已经处于最后阶段,尽管我现在对如何计算标签的概率迷失了方向。到目前为止,我已经完成的工作是统计标签总数,统计单词总数,最后统计每个单词的每个标签。当然还有培训。
对于标签数量,我收到:
{'PRON': 2820, 'VERB': 6201, ...
对于字数统计,我收到:
{'i': 531, 'need': 3, "n't": 213, 'be': 145, ...
对于tagperwordcounts,我收到:
{'i': {'PRON': 531}, 'need': {'VERB': 3}, "n't": {'ADV': 213}, 'be': {'VERB': 145}, 'afraid': {'ADJ': 12}, 'of': {'ADP': 502, 'ADV': 10}, ...
我非常确定我应该用来计算概率的公式是:
(│)=(“标记为”的“的出现次数)/(”“的出现次数
尽管我不确定如何将其放入代码中?
class unigram_tagging():
def __init__(self,traind=[]):
self.tagcounts={}
self.wordcounts={}
self.tagperwordcounts={}
self.train(traind=traind)
def train(self,traind):
for sentence in traind:
for token,tag in sentence:
self.tagcounts[tag]=self.tagcounts.get(tag,0)+1
self.wordcounts[token]=self.wordcounts.get(token,0)+1
current=self.tagperwordcounts.get(token,{})
current[tag]=current.get(tag,0)+1
self.tagperwordcounts[token]=current
def tag(self,traind): #Here I want to work out probability
这个想法是,这种最终方法将分配标签,以最大程度地提高标签概率。