Question

我有一个57m记录和9列的data.table，当我尝试运行一些摘要统计时，其中一个会导致问题。有问题的列是3699级别的因素，我从以下代码行收到错误：

    > unique(da$UPC)
    Error in unique.default(da$UPC): hash table is full

现在很明显我只会使用：levels(da$UPC)但我试图计算每个组中存在的唯一值，作为data.table组语句中多个j参数/ caluclations的一部分。

有趣的是unique(da$UPC[1:1000000])按预期工作，但unique(da$UPC[1:10000000])没有。鉴于我的表有57m记录，这是一个问题。

我尝试将因素转换为字符，但没有问题，如下所示：

    da$UPC = as.character(levels(da$UPC))[da$UPC]
    unique(da$UPC)

这样做确实向我展示了一个额外的“级别”，即NA。因此，因为我的数据在因子列中有一些NA，所以唯一函数无法工作。我想知道这是开发人员知道需要修复的东西吗？我发现以下关于r-devel的文章可能是相关的，但我不确定，也没有提到data.table。

Linked article: unique(1:3,nmax=1) freezes R！

    sessionInfo:

    R version 3.0.1 (2013-05-16)
    Platform: x86_64-unknown-linux-gnu (64-bit)

    locale:
     [1] LC_CTYPE=C                    LC_NUMERIC=C
     [3] LC_TIME=en_US.iso88591        LC_COLLATE=C
     [5] LC_MONETARY=en_US.iso88591    LC_MESSAGES=en_US.iso88591
     [7] LC_PAPER=C                    LC_NAME=C
     [9] LC_ADDRESS=C                  LC_TELEPHONE=C
     [11] LC_MEASUREMENT=en_US.iso88591 LC_IDENTIFICATION=C

    attached base packages:
    [1] stats     graphics  grDevices utils     datasets  methods   base

    other attached packages:
    [1] plyr_1.8         data.table_1.8.8

Answer 1

这段代码应将您缺失的观察结果放入常规级别，以便更易于管理。

# Need additional level to place missing into first
levels(da$UPC) <- c(levels(da$UPC), '(NA)')
da$UPC[is.na(da$UPC)] <- '(NA)'

听起来你最终会尝试降低不频繁的水平以协助进行某种分析。我写了一个函数factorize（），我相信它可以帮到你。它不经常将水平分解为“其他”＃34;类别。

这是链接，如果有帮助，请告诉我。

[比化（）] [1] https://github.com/greenpat/R-Convenience/blob/master/factorize.R

（转载如下）

# This function takes a vector x and returns a factor representation of the same vector.
# The key advantage of factorize is that you can assign levels for infrequent categories,
# as well as empty and NA values. This makes it much easier to perform
# multidimensional/thematic analysis on your largest population subsets.
factorize <- function(
    x,  # vector to be transformed
    min_freq = .01,  # all levels < this % of records will be bucketed
    min_n = 1,  # all levels < this # of records will be bucketed
    NA_level = '(missing)',  # level created for NA values
    blank_level = '(blank)',  # level created for "" values
    infrequent_level = 'Other',  # level created for bucketing rare values
    infrequent_can_include_blank_and_NA = F,  # default NA and blank are not bucketed
    order = T,  # default to ordered
    reverse_order = F  # default to increasing order
) {
    if (class(x) != 'factor'){
        x <- as.factor(x)
    }
    # suspect this is faster than reassigning new factor object
    levels(x) <- c(levels(x), NA_level, infrequent_level, blank_level)

    # Swap out the NA and blank categories
    x[is.na(x)] <- NA_level
    x[x == ''] <- blank_level

    # Going to use this table to reorder
    f_tb <- table(x, useNA = 'always')

    # Which levels will be bucketed?
    infreq_set <- c(
        names(f_tb[f_tb < min_n]),
        names(f_tb[(f_tb/sum(f_tb)) < min_freq])
    )

    # If NA and/or blank were infrequent levels above, this prevents bucketing
    if(!infrequent_can_include_blank_and_NA){
        infreq_set <- infreq_set[!infreq_set %in% c(NA_level, blank_level)]
    }

    # Relabel all the infrequent choices
    x[x %in% infreq_set] <- infrequent_level

    # Return the reordered factor
    reorder(droplevels(x), rep(1-(2*reverse_order),length(x)), FUN = sum, order = order)
}

Answer 2

您可以使用dplyr并获得不同的结果吗？例如，我设置了一些（小）假数据，然后确定alpha的不同级别。我不知道这种情况有多好。

test <- data.frame(alpha=sample(c('a', 'b', 'c'), 100000, replace=TRUE), 
                  num=runif(100000))

uniqueAlpha <- distinct(select(test, alpha))

Answer 3

也许我错过了这一点，但如果它是一个data.table对象，你可以使用它来总结计数：

da[,.N, by=UPC]

如果有效，唯一值为：

unique <- da[,.N, by=UPC]$UPC
length(unique)

您也可以按多列进行分组：

da[,.N,by=.(A,B,C,..)]

Answer 4

不确定它会解决问题，但您可以查看Hadley Wickham的forcats包裹：

library(forcats)
fct_count(da$UPC)

包含NAs =＆gt;的因子列的唯一值“哈希表已满”错误

4 个答案: