将data.table因子列转换为字符列的最快方法

时间:2018-08-15 17:49:10

标签: r data.table

  

TL; DR:将data.table中的所有因子列转换为字符列的最快/最节省内存的方法是什么?

首先,我怀疑这个问题可能会在其他地方得到解答,而我正在努力寻找确定的答案,因此,如果有人能指出正确的方向,我将不胜感激。详尽的答案。

我经常需要将所有data.table因子列转换为字符列。看到Matt Dowle建议像在comment on this answer中一样使用set来完成这项任务时,我以为这是最快/最有效的方式。

但是,当我进行一些测试时,我对结果感到惊讶。方法如下:

测试数据生成

library(data.table)
set.seed(1234)

## Create a vector of 1 million 4 character strings
## with 456,976 possible unique values 
DiverseSize <- 1e6
Diverse <- paste0(sample(LETTERS,DiverseSize,replace = TRUE),
                  sample(letters,DiverseSize,replace = TRUE),
                  sample(letters,DiverseSize,replace = TRUE),
                  sample(letters,DiverseSize,replace = TRUE))

## Create a vector of 10 million single character strings
## with 26 possible unique values
CommonSize  <- 1e7
Common <-  sample(LETTERS,CommonSize,replace = TRUE)

## Mix them into a data.table columns, "x0" through "x9"
DT_Original<- data.table(x0 = as.factor(sample(c(Diverse,Common),size = CommonSize + DiverseSize, replace = FALSE)),
                         x1 = as.factor(sample(c(Diverse,Common),size = CommonSize + DiverseSize, replace = FALSE)),
                         x2 = as.factor(sample(c(Diverse,Common),size = CommonSize + DiverseSize, replace = FALSE)),
                         x3 = as.factor(sample(c(Diverse,Common),size = CommonSize + DiverseSize, replace = FALSE)),
                         x4 = as.factor(sample(c(Diverse,Common),size = CommonSize + DiverseSize, replace = FALSE)),
                         x5 = as.factor(sample(c(Diverse,Common),size = CommonSize + DiverseSize, replace = FALSE)),
                         x6 = as.factor(sample(c(Diverse,Common),size = CommonSize + DiverseSize, replace = FALSE)),
                         x7 = as.factor(sample(c(Diverse,Common),size = CommonSize + DiverseSize, replace = FALSE)),
                         x8 = as.factor(sample(c(Diverse,Common),size = CommonSize + DiverseSize, replace = FALSE)),
                         x9 = as.factor(sample(c(Diverse,Common),size = CommonSize + DiverseSize, replace = FALSE)))

DT1 <- copy(DT_Original)
DT2 <- copy(DT_Original)
DT3 <- copy(DT_Original)

功能

unfactorize <- function(df){
  for(i in which(sapply(df, class) == "factor")) df[[i]] = as.character(df[[i]])
  return(df)
}

set_unfactorize <- function(df){
  for(col in names(df)[which(sapply(df, class) == "factor")]) set(df, j = col, value = as.character(df[[col]]))
}

执行

然后,我使用RStudio内置的性能分析工具来分析以下三个语句。

## Original
DT1 <- unfactorize(DT1)
## data.table::set version
set_unfactorize(DT2)
## Outside of function
for(col in names(DT3)[which(sapply(DT3, class) == "factor")]) set(DT3, j = col, value = as.character(DT3[[col]]))

分析结果

我对结果感到非常惊讶-基本R版本似乎执行得最快并且使用最少的内存,即使我希望它需要一个副本也是如此。是这样吗,还是我在这里错过了什么?

Profiling Comparison

0 个答案:

没有答案