Question

我有一个包含多个列表的文本文件;每个列表都包含单词/ pos-tag对的元组，如下所示：

    [('reviewtext', 'IN'), ('this', 'DT'), ('movie', 'NN'), ('was', 'VBD'), ('great', 'JJ'), ('and', 'CC'), ('fun', 'NN'), ('i', 'PRP'), ('really', 'RB'), ('enjoyed', 'VBD'), ('this', 'DT'), ('awesome', 'NN'), ('movie', 'NN')]
    [('reviewtext', 'IN'), ('it', 'PRP'), ('was', 'VBD'), ('fun', 'VBN'), ('but', 'CC'), ('long', 'RB')]
    [('reviewtext', 'IN'), ('i', 'PRP'), ('loved', 'VBD'), ('the', 'DT'), ('new', 'JJ'), ('movie', 'NN'), ('my', 'PRP$'), ('brother', 'NN'), ('got', 'VBD'), ('sad', 'JJ'), ('and', 'CC'), ('unhappy', 'JJ'), ('at', 'IN'), ('the', 'DT'), ('end', 'NN')]

我需要提取所有形容词 - 连词 - 形容词对或所有JJ-CC-JJ对（仅限单词，而不是pos标签）。对于此示例，最终输出应该是包含所有模式的列表：

    ['great and fun', 'sad and unhappy']

我使用以下代码标记文本：

    with open("C:\\Users\\M\\Desktop\\sample dataset.txt") as fileobject:
        for line in fileobject:
            line = line.lower() #lowercase
            line = re.sub(r'[^\w\s]','',line) #remove punctuation
            line = nltk.word_tokenize(line) #tokenize
            line = nltk.pos_tag(line) #POS tag

            fo = open("C:\\Users\\M\\Desktop\\movies1_complete.txt", "a")
            fo.write(str(line))
            fo.write("\n")
            fo.close()

但是如何提取上述图案中的文字呢？我检查了here和here，但他们没有解释如何提取特定的pos模式。提前谢谢。

Answer 1

from itertools import islice

for sub in l:
    for a, b, c in zip(islice(sub, 0, None), islice(sub, 1, None), islice(sub, 2, None)):
        if all((a[-1] == "JJ", b[-1] == "CC", c[-1] == "JJ")):
            print("{} {} {}".format(a[0], b[0], c[0]))

哪个输出sad and unhappy，它不包含'great and fun'，因为它与模式JJ-CC-JJ不匹配。

或者只使用枚举和生成器：

l = [[('reviewtext', 'IN'), ('this', 'DT'), ('movie', 'NN'), ('was', 'VBD'), ('great', 'JJ'), ('and', 'CC'),
      ('fun', 'NN'), ('i', 'PRP'), ('really', 'RB'), ('enjoyed', 'VBD'), ('this', 'DT'), ('awesome', 'NN'),
      ('movie', 'NN')],
     [('reviewtext', 'IN'), ('it', 'PRP'), ('was', 'VBD'), ('fun', 'VBN'), ('but', 'CC'), ('long', 'RB')],
     [('reviewtext', 'IN'), ('i', 'PRP'), ('loved', 'VBD'), ('the', 'DT'), ('new', 'JJ'), ('movie', 'NN'), ('my', 'PRP$'), ('brother', 'NN'), ('got', 'VBD'), ('sad', 'JJ'), ('and', 'CC'), ('unhappy', 'JJ'), ('at', 'IN'), ('the', 'DT'), ('end', 'NN')]]

def match(l,p1,p2,p3):
    for sub in l:
        # avoid index error and catch last three elements
        end = len(sub) - 1
        for ind, (a, b) in enumerate(sub, 1):
            if ind == end:
                break
            if b == p1 and sub[ind][1] == p2 and sub[ind + 1][1] == p3:
                yield ("{} {} {}".format(a, sub[ind][0], sub[ind + 1][0]))

print(list(match(l,"JJ","CC","JJ")))

输出（基于示例）：

['sad and unhappy']

Answer 2

即使答案已被接受（答案很清楚），我认为你会觉得这很有用。您可以使用以下library来检查对象流中的正则表达式。

from refo import finditer, Predicate, Plus

class Word(object):
    def __init__(self, token, pos):
        self.token = token
        self.pos = pos

class W(Predicate):
    def __init__(self, token=".*", pos=".*"):
        self.token = re.compile(token + "$")
        self.pos = re.compile(pos + "$")
        super(W, self).__init__(self.match)

    def match(self, word):
        m1 = self.token.match(word.token)
        m2 = self.pos.match(word.pos)
        return m1 and m2


originals = [
    [('reviewtext', 'IN'), ('this', 'DT'), ('movie', 'NN'), ('was', 'VBD'), 
     ('great', 'JJ'), ('and', 'CC'), ('fun', 'NN'), ('i', 'PRP'), 
     ('really', 'RB'), ('enjoyed', 'VBD'), ('this', 'DT'), 
     ('awesome', 'NN'), ('movie', 'NN')],
    [('reviewtext', 'IN'), ('it', 'PRP'), 
     ('was', 'VBD'), ('fun', 'VBN'), ('but', 'CC'), ('long', 'RB')],
    [('reviewtext', 'IN'), ('i', 'PRP'), ('loved', 'VBD'), ('the', 'DT'), 
     ('new', 'JJ'), ('movie', 'NN'), ('my', 'PRP$'), ('brother', 'NN'), 
     ('got', 'VBD'), ('sad', 'JJ'), ('and', 'CC'), ('unhappy', 'JJ'), 
     ('at', 'IN'), ('the', 'DT'), ('end', 'NN')]]


sentences = [[Word(*x) for x in original] for original in originals]

这是有趣的一点，它表示查找pos属性为JJ后跟CC后跟JJ或NN的对象序列

pred = W(pos="JJ") + W(pos="CC") + (W(pos="JJ") | W(pos="NN"))
for k, s in enumerate(sentences):
    for match in finditer(pred, s):
        x, y = match.span()   # the match spans x to y inside the sentence s
        print originals[k][x:y]

输出：

[('great', 'JJ'), ('and', 'CC'), ('fun', 'NN')]
[('sad', 'JJ'), ('and', 'CC'), ('unhappy', 'JJ')]

如何从POS标记词列表中提取模式？ NLTK

2 个答案: