Question

输入csv文件：

_id,field_name,field_friendly_name,purpose_of_use,category,data_source,schema,table,attribute_type,sample_values,mask_it,is_included_in_report
5e95a49b0985567430f8fc00,FullName,,,,,,,,,,
5e95a4dd0985567430f9ef16,xyz,,,,,,,,,,
5e95a4dd0985567430f9ef17,FullNm,,,,,,,,,,
5e95a4dd0985567430f9ef18,FirstName,,,,,,,,,,
5e95a49b0985567430f8fc01,abc,,,,,,,,,,
5e95a4dd0985567430f9ef19,FirstNm,,,,,,,,,,
5e95a4dd0985567430f9ef20,LastName,,,,,,,,,,
5e95a4dd0985567430f9ef21,LastNm,,,,,,,,,,
5e95a49b0985567430f8fc02,LegalName,,,,,,,,,,
5e95a4dd0985567430f9ef22,LegalNm,,,,,,,,,,
5e95a4dd0985567430f9ef23,NickName,,,,,,,,,,
5e95a4dd0985567430f9ef24,pqr,,,,,,,,,,
5e95a49b0985567430f8fc03,NickNm,,,,,,,,,,

正则表达式csv表：

Personal_Inforamtion,regex,addiitional_grep
Full Name,full|name|nm|txt|dsc,full
First Name,first|name|nm|txt|dsc,first
Last Name,last|name|nm|txt|dsc,last
Legal Name,legal|name|nm|txt|dsc,legal
Nick Name,nick|name|nm|txt|dsc,nick

我的代码

包括python模块

import pandas as pd
import re

从csv文件定义数据帧

df = pd.read_csv("Default-Profile.csv")

用df替换field_name系列上的下划线（_）和连字符（-）

df.field_name = df.field_name.str.replace("[_-]", "", regex=True)

将df系列中的field_name系列中的所有字符更改为小写

df.field_name = df.field_name.str.lower()

定义正则表达式表

regex_table = pd.read_csv("regex.csv")

代码用于更新field_friendly_name && is_included_in_report

在regex表中为每个正则表达式查找df.field_name中的模式，如果发现正确的匹配项，则使用Personal_information更新字段field_friendly_name；如果未更新为not_found，则将其更新为True；如果发现的匹配项为false，则将最后一列更新为True。

EX：单词应仅由完整的|名称| nm | txt | dsc组成，并且应包含完整的

Personal_Inforamtion,regex,addiitional_grep
Full Name,full|name|nm|txt|dsc,full

然后按如下所示更新df：

_id,field_name,field_friendly_name,purpose_of_use,category,data_source,schema,table,attribute_type,sample_values,mask_it,is_included_in_report
5e95a49b0985567430f8fc00,FullName,Full Name,,,,,,,,,TRUE
5e95a4dd0985567430f9ef16,xyz,not_found,,,,,,,,,FALSE
5e95a4dd0985567430f9ef17,FullNm,Full Name,,,,,,,,,TRUE

所需的输出

_id,field_name,field_friendly_name,purpose_of_use,category,data_source,schema,table,attribute_type,sample_values,mask_it,is_included_in_report
5e95a49b0985567430f8fc00,FullName,Full Name,,,,,,,,,TRUE
5e95a4dd0985567430f9ef16,xyz,not_found,,,,,,,,,FALSE
5e95a4dd0985567430f9ef17,FullNm,Full Name,,,,,,,,,TRUE
5e95a4dd0985567430f9ef18,FirstName,First Name,,,,,,,,,TRUE
5e95a49b0985567430f8fc01,abc,not_found,,,,,,,,,FALSE
5e95a4dd0985567430f9ef19,FirstNm,First Name,,,,,,,,,TRUE
5e95a4dd0985567430f9ef20,LastName,Last Name,,,,,,,,,TRUE
5e95a4dd0985567430f9ef21,LastNm,Last Name,,,,,,,,,TRUE
5e95a49b0985567430f8fc02,LegalName,Legal Name,,,,,,,,,TRUE
5e95a4dd0985567430f9ef22,LegalNm,Legal Name,,,,,,,,,TRUE
5e95a4dd0985567430f9ef23,NickName,NickName,,,,,,,,,TRUE
5e95a4dd0985567430f9ef24,pqr,not_found,,,,,,,,,FALSE
5e95a49b0985567430f8fc03,NickNm,NickName,,,,,,,,,TRUE

Answer 1

或者，您可以创建一组正则表达式来使用正则表文件

(full)|(first)|(last)|(legal)|(nick)

您仍然可以调整regex表的最后一列以获得更具体的输出与您需要。然后，您可以将not_found大小写附加到正则表达式数据框以准备 str.extract使用的数据，该数据从第一个匹配模式中提取组。随着组匹配，然后可以在行轴上使用idxmax获得正则表达式组索引。之后，使用以下命令将正则表达式表第一列上的信息映射到 df 数据框组索引信息。

import pandas as pd
import re

df = pd.read_csv("data.csv")
print(df)

regxt = pd.read_csv("regex_table.csv")
print(regxt)

# append not_found item case
not_found = pd.Series(["not_found","",""], index=regxt.columns)
regxt = regxt.append(not_found, ignore_index=True)

# create regex groups with last column csv words
regxl = regxt.iloc[:, 2].to_list()
regx_grps = "|".join(["(" + i + ")" for i in regxl])

# get regex group match index
grp_match = df["field_name"].str.extract(regx_grps, flags=re.IGNORECASE)
grp_idx = (~grp_match.isnull()).idxmax(axis=1)

df["field_friendly_name"] = grp_idx.map(lambda r: regxt.loc[r, "Personal_Inforamtion"])
df["is_included_in_report"] = grp_idx.map(lambda r: str(r!=len(regxt)-1).upper())

print(df)

df

的输出

                         _id field_name field_friendly_name ... mask_it  is_included_in_report
0   5e95a49b0985567430f8fc00   FullName           Full Name ...     NaN                   TRUE
1   5e95a4dd0985567430f9ef16        xyz           not_found ...     NaN                  FALSE
2   5e95a4dd0985567430f9ef17     FullNm           Full Name ...     NaN                   TRUE
3   5e95a4dd0985567430f9ef18  FirstName          First Name ...     NaN                   TRUE
4   5e95a49b0985567430f8fc01        abc           not_found ...     NaN                  FALSE
5   5e95a4dd0985567430f9ef19    FirstNm          First Name ...     NaN                   TRUE
6   5e95a4dd0985567430f9ef20   LastName           Last Name ...     NaN                   TRUE
7   5e95a4dd0985567430f9ef21     LastNm           Last Name ...     NaN                   TRUE
8   5e95a49b0985567430f8fc02  LegalName          Legal Name ...     NaN                   TRUE
9   5e95a4dd0985567430f9ef22    LegalNm          Legal Name ...     NaN                   TRUE
10  5e95a4dd0985567430f9ef23   NickName           Nick Name ...     NaN                   TRUE
11  5e95a4dd0985567430f9ef24        pqr           not_found ...     NaN                  FALSE
12  5e95a49b0985567430f8fc03     NickNm           Nick Name ...     NaN                   TRUE

使用熊猫数据框的正则表达式

1 个答案: