NLTK fcfg语法分析器超出索引

时间:2018-01-10 20:31:03

标签: python nltk

我是NLTK的新手。试图改变"给我看电影"进入一个简单的SQL SELECT语句"从电影"中选择标题。

我认为句子是(VP + NP),VP(V + PRO)和NP(DET + N)。但是我毫不怀疑我设置的.fcfg语法是不正确的,我在" anwser = trees"上得到以下索引错误,树是空的。

如何更正.fcfg?

  

IndexError:列表索引超出范围

     

使用退出代码1完成处理

分析器

% start S
S[SEM=(?np + WHERE + ?vp)] -> NP[SEM=?np] VP[SEM=?vp]
VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> Det[SEM=?det] N[SEM=?n]
Det[SEM=''] -> 'the'
PRO[SEM=''] -> 'me'
N[SEM='title FROM films'] -> 'movies'
V[SEM='SELECT'] -> 'show'

Python代码

from nltk import load_parser
cp = load_parser('parser3.fcfg')
query = 'show me the movies'
trees = list(cp.parse(query.split()))
print(trees)
answer = trees[0].label()['SEM']
answer = [s for s in answer if s]
q = ' '.join(answer)
print(q)

1 个答案:

答案 0 :(得分:0)

要调试语法,请从小处开始并扩展规则。

让我们从未指定的VP和功能结构的V和N

开始
from nltk import grammar, parse
from nltk.parse.generate import generate

g = """
VP -> V N
V[SEM='SELECT'] -> 'show'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show movies'.split())
print (list(trees)) 

[OUT]:

[Tree(VP[], [Tree(V[SEM='SELECT'], ['show']), Tree(N[SEM='title FROM films'], ['movies'])])]

现在,让我们添加限定词。

g = """
VP -> V NP
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]
DT[SEM=''] -> 'the'
V[SEM='SELECT'] -> 'show'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show the movies'.split())
print (list(trees)) 

[OUT]:

[Tree(VP[], [Tree(V[SEM='SELECT'], ['show']), Tree(NP[SEM=(, title FROM films)], [Tree(DT[SEM=''], ['the']), Tree(N[SEM='title FROM films'], ['movies'])])])]

然后我们添加代词。

我们要解析“给我看电影”这句话

S[ VP[show me] NP[the movie] ]

所以我们必须改变我们的TOP到S -> VP NP的方式。

g = """
S -> VP NP
VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]
V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

[OUT]:

[Tree(S[], [Tree(VP[SEM=(SELECT, )], [Tree(V[SEM='SELECT'], ['show']), Tree(PRO[SEM=''], ['me'])]), Tree(NP[SEM=(, title FROM films)], [Tree(DT[SEM=''], ['the']), Tree(N[SEM='title FROM films'], ['movies'])])])]

这就是神秘的

我们的TOP规则目前尚未明确规定,但如果我们同时指定左侧(LHS)和右侧(RHS),我们会发现它不起作用:

g = """
S[SEM=(?vp + WHERE + ?np)] -> VP[SEM=?vp] NP[SEM=?np]

VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]

V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

即使我们删除了WHERE语义,我们也发现它没有解析:

g = """
S[SEM=(?vp + ?np)] -> VP[SEM=?vp] NP[SEM=?np]

VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]

V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

[OUT]:

[]

但是如果我们只指定RHS,它会解析:

g = """
S -> VP[SEM=?vp] NP[SEM=?np]

VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]

V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

[OUT]:

[Tree(S[], [Tree(VP[SEM=(SELECT, )], [Tree(V[SEM='SELECT'], ['show']), Tree(PRO[SEM=''], ['me'])]), Tree(NP[SEM=(, title FROM films)], [Tree(DT[SEM=''], ['the']), Tree(N[SEM='title FROM films'], ['movies'])])])]

当我们仅指定LHS时,同样有效:

g = """
S[SEM=(?vp + WHERE + ?np)] -> VP NP

VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]

V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

[OUT]:

[Tree(S[SEM=(?vp+WHERE+?np)], [Tree(VP[SEM=(SELECT, )], [Tree(V[SEM='SELECT'], ['show']), Tree(PRO[SEM=''], ['me'])]), Tree(NP[SEM=(, title FROM films)], [Tree(DT[SEM=''], ['the']), Tree(N[SEM='title FROM films'], ['movies'])])])]

让我们回顾一下

我们可以像NP和VP一样指定非终端,但是TOP(即S -> VP NP)的特殊之处是什么?

如果我们破解语法并且只是从一个单一的分支出来怎么办?

g = """
S -> SP
SP[SEM=(?vp + WHERE + ?np)] -> VP[SEM=?vp] NP[SEM=?np]

VP[SEM=(?v + ?pro)] -> V[SEM=?v] PRO[SEM=?pro]
NP[SEM=(?det + ?n)] -> DT[SEM=?det] N[SEM=?n]

V[SEM='SELECT'] -> 'show'
PRO[SEM=''] -> 'me'
DT[SEM=''] -> 'the'
N[SEM='title FROM films'] -> 'movies'
"""

my_grammar =  grammar.FeatureGrammar.fromstring(g)

parser = parse.FeatureEarleyChartParser(my_grammar)
trees = parser.parse('show me the movies'.split())
print (list(trees)) 

解析了!

[OUT]:

[Tree(S[], [Tree(SP[SEM=(SELECT, , WHERE, , title FROM films)], [Tree(VP[SEM=(SELECT, )], [Tree(V[SEM='SELECT'], ['show']), Tree(PRO[SEM=''], ['me'])]), Tree(NP[SEM=(, title FROM films)], [Tree(DT[SEM=''], ['the']), Tree(N[SEM='title FROM films'], ['movies'])])])])]

有人应该向NLTK github repo提出这个问题。看起来它可能是保护TOP规则或bug =)的特殊功能