Question

我正在尝试在python 3中使用spacy从文本中提取语言特征。我的输入看起来像这样

Sent_id Text
1   I am exploring text analytics using spacy
2   amazing spacy is going to help me

我正在通过将单词提取为具有我提供的特定POS模式的三语组/二语组短语来寻找这样的输出。像名词VERB名词，ADJ名词等，并保留数据框结构。如果一个句子中有多个词组，则该记录必须与新词组重复。

Sent_id Text    Feature Pattern
1   I am exploring text analytics using spacy   exploring text analytics    VERB NOUN NOUN
1   I am exploring text analytics using spacy   analytics using spacy   NOUN VERB NOUN
2   amazing spacy is going to help me   amazing spacy   ADJ NOUN

Answer 1

intput: 1234500000 output: 2345000000内置了强大的模式匹配语言：spaCy docs和SO answer用于解决类似问题。

通过这种方式，您不必像accepted answer中的@mujjiga一样将正则表达式和POS标签混在一起。

所以您的示例如下所示：

spacy

打印出3个匹配项及其范围（开始和结束单词位置）：

nlp = spacy.load('en_core_web_sm')
from spacy.matcher import Matcher

examples = [
    "I am exploring text analytics using spacy",
    "amazing spacy is going to help me",
    ]
patterns = [
    [{'POS': 'NOUN'}, {'POS': 'VERB'}, {'POS': 'NOUN'}],
    [{'POS': 'VERB'}, {'POS': 'NOUN'}],
    ]
matcher = Matcher(nlp.vocab)
matcher.add("noun-verb-noun", None, patterns[1])
matcher.add("verb-noun", None, patterns[0])
for text in examples:
    doc = nlp(text)
    matches = matcher(doc)
    for match_id, start, end in matches:
        string_id = nlp.vocab.strings[match_id]
        span = doc[start:end]
        print(string_id, start, end, span.text)

Answer 2

代码在注释中说明

import spacy
import pandas as pd
import re

# Load spacy model once and reuse 
nlp = spacy.load('en_core_web_sm')

# The dataframe with text
df = pd.DataFrame({
        'Sent_id': [1,2],
        'Text': [ "I am exploring text analytics using spacy", "amazing spacy is going to help me"]
    }) 

# Patters we are intrested in 
patterns = ["VERB NOUN", "NOUN VERB NOUN"]

# Convert each pattern into regular expression
re_patterns = [" ".join(["(\w+)_!"+pos for pos in p.split()]) for p in patterns]


def extract(nlp, text, patterns, re_patterns):
    """Extracts the pieces in text maching the POS pattern in patterns

    args:
        nlp : Loaded Spicy model object
        text: The input text
        patterns: The list of patters to be searched
        re_patterns: The patterns converted into regex

    returns: A list of tuples of form (t,p) where 
    t is the part of text matching the pattern p in patterns
    """
    doc = nlp(text)   
    matches = list()
    text_pos = " ".join([token.text+"_!"+token.pos_ for token in doc])
    for i, pattern in enumerate(re_patterns):
        for result in re.findall(pattern, text_pos):
            matches.append([" ".join(result), patterns[i]])
    return matches

# Test it 
print (extract(nlp, "A sleeping cat and walking dog", patterns, re_patterns))
# Returns
# [['sleeping cat', 'VERB NOUN'], ['walking dog', 'VERB NOUN']]

# Extract the matched patterns
df['matches'] = df['Text'].apply(lambda x: extract(nlp,x,patterns,re_patterns))


# Convert the list of tuples into rows
df = df.matches.apply(pd.Series).merge(df, right_index = True, left_index = True).drop(["matches"], axis = 1)\
.melt(id_vars = ['Sent_id', 'Text'], value_name = "matches").drop("variable", axis = 1)

# Add the matched text and matched patterns into new columns
df[['matched_text','matched_pattern']]= df.matches.apply(pd.Series)

# Drop the column and cleanup
df = df.drop("matches", axis = 1).sort_values('Sent_id')
df = df.drop_duplicates(subset =["matched_text", "matched_pattern"], keep='last')

输出：

    Sent_id     Text                                matched_text    matched_pattern
0   1   I am exploring text analytics using spacy   exploring text  VERB NOUN
2   1   I am exploring text analytics using spacy   using spacy     VERB NOUN
4   1   I am exploring text analytics using spacy   analytics using spacy   NOUN VERB NOUN
1   2   amazing spacy is going to help me           NaN              NaN

具有spacy的POS模式挖掘

2 个答案:

代码在注释中说明

输出：