Question

我对R中的代码有疑问。

我有一个包含4列和超过600k观察的数据集（问题），其中一列被命名为＆＃39; V3＆＃39;。本专栏有类似“今天是什么日子”的问题？＆＃39;。我有第二个数据集（voc）有2列，其中一列名称＆＃39; word＆＃39;和其他列名＆＃39;同义词＆＃39;。如果在我的第一个数据集（问题）中存在来自第二个数据集（voc）的单词＆＃39;同义词＆＃39;然后我想用＆＃39; word＆＃39;替换它。列。

questions = cbind(V3=c("What is the day today?","Tom has brown eyes"))
questions <- data.frame(questions)  

                      V3                                                                                            
1 what is the day today?                                                                                             
2     Tom has brown eyes  

voc = cbind(word=c("weather", "a","blue"),synonyms=c("day", "the", "brown"))
voc <- data.frame(voc)

     word synonyms                                                                                                    
1 weather      day                                                                                               
2       a      the                                                                                                   
3    blue    brown 

Desired output

                      V3                        V5                                                                                  
1 what is the day today?  what is a weather today?                                                                                          
2     Tom has brown eyes         Tom has blue eyes

我编写了简单的代码，但它不起作用。

for (k in 1:nrow(question))
{
    for (i in 1:nrow(voc))
   {
      question$V5<- gsub(do.call(rbind,strsplit(question$V3[k]," "))[which (do.call(rbind,strsplit(question$V3[k]," "))== voc[i,2])], voc[i,1], question$V3)
   }
}

也许有人会试着帮助我？：）

我写了第二段代码，但它也没有用。

for( i in 1:nrow(questions))
{
    for( j in 1:nrow(voc))
      {
        if (grepl(voc[j,k],do.call(rbind,strsplit(questions[i,]," "))) == TRUE)
        {
            new=matrix(gsub(do.call(rbind,strsplit(questions[i,]," "))[which(do.call(rbind,strsplit(questions[i,]," "))== voc[j,2])], voc[j,1], questions[i,]))
            questions[i,]=new   
        }
    }
    questions = cbind(questions,c(new))
}

Answer 1

首先，您必须在程序级别或数据导入期间使用stringsAsFactors = FALSE选项。这是因为除非另有说明，否则R默认将字符串作为因子。因素在建模中非常有用，但您希望对文本本身进行分析，因此您应该确保文本不会被强制转换为因素。

我接近这个的方法是编写一个函数，将每个字符串“爆炸”成一个向量，然后使用匹配来替换这些单词。矢量再次重新组合成一个字符串。

我不确定这会给你的600K记录带来多大的好处。您可能会查看一些处理字符串的R包，例如stringr或stringi，因为它们可能具有执行此操作的函数。 match在速度方面往往没问题，但%in%可能是真正的野兽，具体取决于字符串的长度和其他因素。

# Start with options to make sure strings are represented correctly
# The rest is your original code (mildly tidied to my own standard)
options(stringsAsFactors = FALSE)
questions <- cbind(V3 = c("What is the day today?","Tom has brown eyes"))
questions <- data.frame(questions)  

voc <- cbind(word = c("weather","a","blue"),
             synonyms = c("day","the","brown"))
voc <- data.frame(voc)

# This function takes:
#  - an input string
#  - a vector of words to replace
#  - a vector of the words to use as replacements
# It returns a list of the original input and the changed version    
uFunc_FindAndReplace <- function(input_string,words_to_repl,repl_words) {

    # Start by breaking the input string into a vector
    # Note that we use [[1]] to get first list element of strsplit output
    # Obviously this relies on breaking sentences by spacing
    orig_words <- strsplit(x = input_string,split = " ")[[1]]

    # If we find at least one of the words to replace in the original words, proceed
    if(sum(orig_words %in% words_to_repl) > 0) {

        # The right side selects the elements of orig_words that match words to be replaced
        # The left side uses match to find the numeric index of those replacements within the words_to_repl vector
        # This numeric vector is used to select the values from repl_words
        # These then replace the values in orig_words
        orig_words[orig_words %in% words_to_repl] <- repl_words[match(x = orig_words,table = words_to_repl,nomatch = 0)]

        # We rebuild the sentence again, and return a list with original and new version
        new_sent <- paste(orig_words,collapse = " ")
        return(list(original = input_string,new = new_sent))
    } else {

        # Otherwise we return the original version since no changes are needed
        return(list(original = input_string,new = input_string))
    }
}

# Using do.call and rbind.data.frame, we can collapse the output of a lapply()

do.call(what = rbind.data.frame,
        args = lapply(X = questions$V3,
                      FUN = uFunc_FindAndReplace,
                      words_to_repl = voc$synonyms,
                      repl_words = voc$word))

>
                original                      new
1 What is the day today? What is a weather today?
2     Tom has brown eyes        Tom has blue eyes

NLP - 识别和替换R中的单词（同义词）

1 个答案: