通过匹配和替换对段落变量中的名称进行匿名化

时间:2016-09-05 20:22:55

标签: r tm reproducible-research data-scrubbing anonymize

我正在分析学校的学生成绩单数据库。我的数据集包含大约3000条记录,其结构与下面的示例类似。每次观察都是一位教师对一名学生的评估。每个观察都包含一个三句话的叙述评论。

为了分享我的分析结果,我想从评论中提取学生姓名,并将其替换为其他名称。在理想的世界中,我还想分享一个匿名版本的数据库,以便重现。

学生姓名的不一致使用(首先是昵称与全名)和非结构化使用学生的名字对于像我这样的业余爱好者来说这非常棘手。我尝试解决这个问题的方法是将注释作为语料库中的文档进行处理,并使用编写一个使用tm::removeWords的函数,但它对我来说并不起作用。提前谢谢!

示例数据(dput of table here)

  Teacher Subject      Student.Name                                                         Comment
1   Black    Math    Richard (Dick) Dick is a terrible student-- why hasn't he been kicked out yet?
2   Black    Math Elizabeth (Betty)                       Betty procrastinates, but does good work.
3   Black    Math   Mary Grace (MG)                      As her teacher, I think MG is my favorite.
4   Brown English    Richard (Dick)                      Richard is terrible at turning in homework.
5   Brown English Elizabeth (Betty)                Elizabeth's work is interfering with her studies.
6   Brown English   Mary Grace (MG)                         Mary Grace should be a teacher someday.
7    Blue    P.E.    Richard (Dick)  Richard (Dick) kicked more field goals than any other student.
8    Blue    P.E. Elizabeth (Betty)    Elizabeth (Betty) needs to work to communicate on the field.
9    Blue    P.E.   Mary Grace (MG)             Mary Grace (MG) needs to stop insulting the teacher

所需数据

Teacher Subject Student Name    Comment
Black   Math    A   A is a terrible student-- why hasn't he been kicked out yet?
Black   Math    B   B procrastinates, but does good work.
Black   Math    C   As her teacher, I think C is my favorite.
Brown   English A   A is terrible at turning in homework
Brown   English B   B's work is interfering with her studies.
Brown   English C   C should be a teacher someday.
Blue    P.E.    A   A kicked more field goals than any other student.
Blue    P.E.    B   B needs to work to communicate on the field.
Blue    P.E.    C   C needs to stop insulting the teacher

N.B。

四个月前,我要求a version of this question没有回复。我认为这有助于展示我的解决方案但是tm包可能没有被广泛使用。所以这是另一个镜头。

2 个答案:

答案 0 :(得分:2)

我会在mgsub包中使用qdap。你可以做这样的事情(尽管要注意确保学生归因于同样的ID,这些ID可能对你的例子过于具体,其中包含每个学生的昵称):

names <- unique(as.character(reports$Student.Name))
ids <- sample(100000, length(names))

tocheck <- c(
  names, 
  unlist(regmatches(names, gregexpr("(?<=\\().*?(?=\\))", names, perl = T))),
  gsub("\\s*\\([^\\)]+\\)","",as.character(names))
)
reports$Student.Name <- rep(ids, 3)
reports$Comment <- qdap::mgsub(tocheck, rep(ids, 3), reports$Comment)

  Student.Name                                                          Comment
1        61034 61034 is a terrible student-- why hasn't he been kicked out yet?
2        45005                        45005 procrastinates, but does good work.
3        13699                    As her teacher, I think 13699 is my favorite.
4        61034                         61034 is terrible at turning in homework
5        45005                    45005's work is interfering with her studies.
6        13699                               13699 should be a teacher someday.
7        61034            61034 kicked more field goals than any other student.
8        45005                 45005 needs to work to communicate on the field.
9        13699                        13699 needs to stop insulting the teacher

答案 1 :(得分:1)

我认为这并不是一个简单的“一刀切”的解决方案。我可能会尝试正则表达式。

## load dput data
#eval(parse(text=paste0(readLines("http://pastebin.com/raw/MbghGybd", warn = F), collapse="\n")))

# anonymize:
r <-  regexec("(\\w+)\\s(?:(\\w+)\\s)?\\((\\w+)\\)", levels(reports$Student.Name))
m <- regmatches(levels(reports$Student.Name), r)
names(m) <- levels(reports$Student.Name)
m <- lapply(m, function(x) { 
  paste(sprintf("%s\\s*\\(%s\\)", x[2], x[4]), sprintf("%s %s \\(%s\\)", x[2], x[3], x[4]), x[2], x[4], paste(x[2], x[3], sep=" "), sep="|")
})
rep <- split(reports, reports$Student.Name)
for (x in seq_along(names(rep))) {
  rep[[x]]$Comment <-  gsub(m[[names(rep)[x]]], x, rep[[x]]$Comment, perl=TRUE)
}
transform(do.call(rbind, rep), Student.Name=as.integer(Student.Name))
#                     Teacher Subject Student.Name                                                      Comment
# Elizabeth (Betty).2   Black    Math            1                        1 procrastinates, but does good work.
# Elizabeth (Betty).5   Brown English            1                    1's work is interfering with her studies.
# Elizabeth (Betty).8    Blue    P.E.            1                 1 needs to work to communicate on the field.
# Mary Grace (MG).3     Black    Math            2                    As her teacher, I think 2 is my favorite.
# Mary Grace (MG).6     Brown English            2                         2 Grace should be a teacher someday.
# Mary Grace (MG).9      Blue    P.E.            2                        2 needs to stop insulting the teacher
# Richard (Dick).1      Black    Math            3 3 is a terrible student-- why hasn't he been kicked out yet?
# Richard (Dick).4      Brown English            3                         3 is terrible at turning in homework
# Richard (Dick).7       Blue    P.E.            3            3 kicked more field goals than any other student.

但这肯定需要进行大量调整才能使您的真实数据集成形。