Question

我有一个包含距离的数据表。我希望通过我的“id”变量和包含距离阈值(e.g. Dist<1, Dist<2, etc.).在data.table中运行各种操作。我知道如何通过id和距离"by=list(id,Dist)"运行操作，但我真的想要一个by变量更像是"by=list(id,c(Dist<=1,Dist<=2,Dist<=3,Dist<=4,Dist<=5)。以下是我的数据结构和目标的示例。

#load library
    library(data.table)
#create data  
    set.seed(123L)
    dt<-data.table(id=factor(rep(1:10,5)),V1=rnorm(50,5,5),Dist=sample(1:5,50,replace=T))
#calculate mean of V1 by id and distance (wrong results)
    dt2<-dt[,.(MeanV1=mean(V1)),by=list(id,Dist)]
#calculate mean of V1 by id and conditional distance (right results, wrong method)
    dt2.1<-dt[Dist<=1,.(MeanV1=mean(V1)),by=id]
    dt2.2<-dt[Dist<=2,.(MeanV1=mean(V1)),by=id]
    dt2.3<-dt[Dist<=3,.(MeanV1=mean(V1)),by=id]
    dt2.4<-dt[Dist<=4,.(MeanV1=mean(V1)),by=id]
    dt2.5<-dt[Dist<=5,.(MeanV1=mean(V1)),by=id]
    dt2<-rbind(dt2.1,dt2.2,dt2.3,dt2.4,dt2.5)
#ideal methods if either were valid
    #syntax 1
        dt2<-dt[,.(MeanV1=mean(V1)),by=list(id,c(Dist<=1,Dist<=2,Dist<=3,Dist<=4,Dist<=5))]
    #syntax 2
        rowindices<-list(dt$Dist<=1,dt$Dist<=2,dt$Dist<=3,dt$Dist<=4,dt$Dist<=5)
        dt2<-dt[,.(MeanV1=mean(V1)),by=list(id,rowindices)]

提前致谢。

Answer 1

弗兰克在评论中的回答将实现你所追求的目标。这是一个解释：

首先，你可以用data.table做的一件事是“非equi join”，这是第一个data.table调用正在做的事情。

首先，我们创建一个我们想要操作的阈值表：

> thresholds <- data.table(dist_threshold=1:5)
> thresholds
   dist_threshold
1:              1
2:              2
3:              3
4:              4
5:              5

接下来，我们使用阈值表在原始表上执行非equi连接：这将创建一个新表，其中dist列包含低于该阈值的每个ID的所有条目：

> passes_threshold <- dt[thresholds, on=.(Dist < dist_threshold), # non-equi join
+    allow.cartesian=TRUE, # Fixes error, see details in ?data.table
+    nomatch=0 # Do not include thresholds which no row satisfies (i.e. Dist < 1)
+   ]
> passes_threshold
# Here the Dist column now means "Dist < dist_threshold".
# There will be 5 rows where Dist < 2, 19 where Dist < 3, 
# 30 where Dist < 4, and 40 Where Dist < 5 
    id        V1 Dist
 1:  8  8.521825    2
 2:  5  2.002523    2
 3:  6  8.698732    2
 4:  9 -1.701028    2
 5:  2  6.114119    2
---                  
90:  6 -1.392776    5
91: 10  9.033493    5
92:  1  9.565713    5
93:  5  4.579124    5
94:  7  1.498690    5

我们现在可以将联接与j和by参数中的汇总操作结合起来，计算每个阈值的平均距离：

> passes_threshold[,.(mean_Dist_by_threshold=mean(V1)), by=.(threshold=Dist)]
       threshold mean_Dist_per_threshold
1:         2                4.727234
2:         3                4.615258
3:         4                4.202856
4:         5                4.559240

Answer 2

作为Scott's answer的补充，他的解决方案可以更简洁地编写为

dt[.(1:5), on = .(Dist < V1), allow = TRUE, nomatch = 0][
  , .(mean_Dist_by_threshold = mean(V1)), by = .(threshold = Dist)]

此处，.(1:5)动态创建thresholds ，data.table表达式被链接。

或者，可以使用by = .EACHI在加入期间完成聚合：

dt[.(1:5), on = .(Dist < V1), nomatch = 0, 
   .(mean_Dist_by_threshold = mean(V1)), by = .EACHI][
     , setnames(.SD, "Dist", "threshold")]

对setnames()的调用只是为了方便返回与Scott的答案相同的结果。

基准代码

library(data.table)
# create data
nr <- 5e2L
set.seed(123L) # to make the data reproducible
dt <-
  data.table(
    id = factor(rep(1:10, nr / 10)),
    V1 = rnorm(nr, 5, 5),
    Dist = sample(1:5, nr, replace = T)
  )
str(dt)

microbenchmark::microbenchmark(
  scott = {
    thresholds <- data.table(dist_threshold=1:5)
    passes_threshold <-
      dt[thresholds, on = .(Dist < dist_threshold), # non-equi join
         allow.cartesian = TRUE, # Fixes error, see details in ?data.table
         nomatch = 0 # Do not include thresholds which no row satisfies (i.e. Dist < 1)
         ]
    passes_threshold[, .(mean_Dist_by_threshold = mean(V1)), by = .(threshold = Dist)]
  },
  uwe1 = {
    dt[.(1:5), on = .(Dist < V1), allow = TRUE, nomatch = 0][
      , .(mean_Dist_by_threshold = mean(V1)), by = .(threshold = Dist)]
  },
  uwe2 = {
    dt[.(1:5), on = .(Dist < V1), nomatch = 0, 
       .(mean_Dist_by_threshold = mean(V1)), by = .EACHI][
         , setnames(.SD, "Dist", "threshold")]
  },
  times = 100L
)

基准测试结果

有500行，3种变体之间只有轻微差异，链接略高于斯科特和by = .EACHI。

Unit: milliseconds
  expr      min       lq     mean   median       uq       max neval cld
 scott 1.460058 1.506854 1.618048 1.526019 1.726257  4.768493   100  a 
  uwe1 1.302760 1.327686 1.487237 1.338926 1.372498 12.733933   100  a 
  uwe2 1.827756 1.864777 1.944920 1.888349 2.020097  2.233269   100   b

有50000行，链接仍然略高于斯科特，但by = .EACHI的表现优于其他链接。

Unit: milliseconds
  expr      min       lq     mean   median       uq       max neval cld
 scott 3.692545 3.811466 4.016152 3.826423 3.853489 10.336598   100   b
  uwe1 3.560786 3.632999 3.936583 3.642526 3.657992 13.579008   100   b
  uwe2 2.503508 2.545722 2.577735 2.566869 2.602586  2.798692   100  a

有5 M行，这变得更加明显：

Unit: milliseconds
  expr      min       lq     mean   median       uq      max neval cld
 scott 641.9945 675.3749 743.0761 708.7552 793.6170 878.4787     3   b
  uwe1 587.1724 587.5557 589.1360 587.9391 590.1178 592.2965     3   b
  uwe2 130.9358 134.6688 157.1860 138.4019 170.3110 202.2202     3  a

速度差异的一个解释可能是超过10 M行的中间结果passes_threshold的剪切大小（这就是为什么需要allow.cartesian = TRUE）。

R data.table由条件或行索引列表组成

2 个答案:

基准代码

基准测试结果