Python Pandas专栏和模糊匹配+替换

Question

简介

您好，我正在开发一个项目，要求我用值替换文本的pandas列中的字典键 - 但是可能会出现拼写错误。具体来说，我匹配文本的pandas列中的名称，并将其替换为“First Name”。例如，我将用“名字”取代“tommy”。

但是，我意识到字符串列中的拼写错误的名称和文本问题不会被我的字典替换。例如'tommmmy'有额外的m并且不是我字典中的第一个名字。

#Create df 
d = {'message' : pd.Series(['awesome', 'my name is tommmy , please help with...', 'hi tommy , we understand your quest...'])}
names = ["tommy", "zelda", "marcon"]

#create dict 
namesdict = {r'(^|\s){}($|\s)'.format(el): r'\1FirstName\2' for el in names}

#replace 
d['message'].replace(namesdict, regex = True)



  #output 
    Out: 
0                                       awesome
1    my name is tommmy , please help with...
2    hi FirstName , we understand your quest...
dtype: object

所以“tommmy”与 - ＆gt;中的“tommy”不匹配我需要处理拼写错误。我想在实际的字典键和值替换之前尝试这样做，比如扫描pandas数据帧并用适当的名称替换字符串列（“messages”）中的单词。我见过类似的方法，使用特定字符串的索引，如this one

但如何使用正确的拼写列表匹配并替换pandas df中句子中的单词？我可以在df.series replace参数中执行此操作吗？我应该坚持使用正则表达式字符串替换吗？*

任何建议表示赞赏。

更新，尝试Yannis的回答

我正在尝试Yannis的答案，但我需要使用外部来源的列表，特别是美国人口普查的匹配名称。但它与我下载的字符串在整个名称上不匹配。

d = {'message' : pd.Series(['awesome', 'my name is tommy , please help with...', 'hi tommy , we understand your quest...'])}

import requests 
r = requests.get('http://deron.meranda.us/data/census-derived-all-first.txt')

#US Census first names (5000 +) 
firstnamelist = re.findall(r'\n(.*?)\s', r.text, re.DOTALL)


#turn list to string, force lower case
fnstring = ', '.join('"{0}"'.format(w) for w in firstnamelist )
fnstring  = ','.join(firstnamelist)
fnstring  = (fnstring.lower())


##turn to list, prepare it so it matches the name preceded by either the beginning of the string or whitespace.  
names = [x.strip() for x in fnstring.split(',')]




#import jellyfish 
import difflib 


def best_match(tokens, names):
    for i,t in enumerate(tokens):
        closest = difflib.get_close_matches(t, names, n=1)
        if len(closest) > 0:
            return i, closest[0]
    return None

def fuzzy_replace(x, y):

    names = y # just a simple replacement list
    tokens = x.split()
    res = best_match(tokens, y)
    if res is not None:
        pos, replacement = res
        tokens[pos] = "FirstName"
        return u" ".join(tokens)
    return x

d["message"].apply(lambda x: fuzzy_replace(x, names))

结果：

Out: 
0                                        FirstName
1    FirstName name is tommy , please help with...
2    FirstName tommy , we understand your quest...

但是，如果我使用这样的较小列表，它可以工作：

names = ["tommy", "caitlyn", "kat", "al", "hope"]
d["message"].apply(lambda x: fuzzy_replace(x, names))

是否存在导致问题的较长名单列表？

Answer 1

修改

更改了我的解决方案以使用difflib。核心思想是对输入文本进行标记，并将每个标记与名称列表进行匹配。如果best_match找到匹配项，则会报告位置（以及最匹配的字符串），因此您可以使用＆＃34; FirstName＆＃34;或任何你想要的。请参阅下面的完整示例：

import pandas as pd
import difflib

df = pd.DataFrame(data=[(0,"my name is tommmy , please help with"), (1, "hi FirstName , we understand your quest")], columns=["A", "message"])

def best_match(tokens, names):
    for i,t in enumerate(tokens):
        closest = difflib.get_close_matches(t, names, n=1)
        if len(closest) > 0:
            return i, closest[0]
    return None

def fuzzy_replace(x):
    names = ["tommy", "john"] # just a simple replacement list
    tokens = x.split()
    res = best_match(tokens, names)
    if res is not None:
        pos, replacement = res
        tokens[pos] = "FirstName"
        return u" ".join(tokens)
    return x

df.message.apply(lambda x: fuzzy_replace(x))

你应该得到的输出如下

0    my name is FirstName , please help with
1    hi FirstName , we understand your quest
Name: message, dtype: object

修改2

在讨论之后，我决定再次使用NLTK进行词性标注，并仅针对名称列表运行NNP标签（专有名词）的模糊匹配。问题是有时标记器没有正确标记，例如＆＃34;您好＆＃34;也可能被标记为专有名词。但是，如果名称列表是小写的，则get_close_matches与名称匹配Hi但与所有其他名称匹配。我建议df["message"]不要小写，以增加NLTK正确标记名称的机会。人们也可以和斯坦福纳一起玩，但没有任何东西可以100％运作。这是代码：

import pandas as pd
import difflib
from nltk import pos_tag, wordpunct_tokenize
import requests 
import re

r = requests.get('http://deron.meranda.us/data/census-derived-all-first.txt')

# US Census first names (5000 +) 
firstnamelist = re.findall(r'\n(.*?)\s', r.text, re.DOTALL)

# turn list to string, force lower case
# simplified things here
names = [w.lower() for w in firstnamelist]


df = pd.DataFrame(data=[(0,"My name is Tommmy, please help with"), 
                        (1, "Hi Tommy , we understand your question"),
                        (2, "I don't talk to Johhn any longer"),
                        (3, 'Michale says this is stupid')
                       ], columns=["A", "message"])

def match_names(token, tag):
    print token, tag
    if tag == "NNP":
        best_match = difflib.get_close_matches(token, names, n=1)
        if len(best_match) > 0:
            return "FirstName" # or best_match[0] if you want to return the name found
        else:
            return token
    else:
        return token

def fuzzy_replace(x):
    tokens = wordpunct_tokenize(x)
    pos_tokens = pos_tag(tokens)
    # Every token is a tuple (token, tag)
    result = [match_names(token, tag) for token, tag in pos_tokens]
    x = u" ".join(result)
    return x

df['message'].apply(lambda x: fuzzy_replace(x))

我进入输出：

0       My name is FirstName , please help with
1    Hi FirstName , we understand your question
2        I don ' t talk to FirstName any longer
3                 FirstName says this is stupid
Name: message, dtype: object