在为问题answer准备dplyr or data.table to calculate time series aggregations in R时,我注意到我确实得到了不同的结果,具体取决于表是在适当的位置更新还是作为新对象返回。此外,当我更改 non-equi join 条件中的列顺序时,我会得到不同的结果。
目前,我没有对此做出解释,可能是由于我身边的一个重大误解或一个简单的编码错误。
请注意,这个问题特别要求 对
data.table
连接的观察行为的解释。如果你 对于潜在问题有其他解决方案,拜托,感觉 免费发布original question的答案。
最初的问题是如何使用这些数据计算每位患者在住院前365天(包括实际住院期间)住院次数:
library(data.table) # version 1.10.4 (CRAN) or 1.10.5 (devel built 2017-08-19)
DT0 <- data.table(
patient.id = c(1L, 2L, 1L, 1L, 2L, 2L, 2L),
hospitalization.date = as.Date(c("2013/10/15", "2014/10/15", "2015/7/16", "2016/1/7",
"2015/12/20", "2015/12/25", "2016/2/10")))
setorder(DT0, patient.id, hospitalization.date)
DT0
patient.id hospitalization.date 1: 1 2013-10-15 2: 1 2015-07-16 3: 1 2016-01-07 4: 2 2014-10-15 5: 2 2015-12-20 6: 2 2015-12-25 7: 2 2016-02-10
下面的代码给出了预期答案(为清楚起见,此处添加了额外的辅助列)
# add helper columns
DT0[, start.date := hospitalization.date - 365][
, end.date := hospitalization.date][]
DT0
patient.id hospitalization.date start.date end.date 1: 1 2013-10-15 2012-10-15 2013-10-15 2: 1 2015-07-16 2014-07-16 2015-07-16 3: 1 2016-01-07 2015-01-07 2016-01-07 4: 2 2014-10-15 2013-10-15 2014-10-15 5: 2 2015-12-20 2014-12-20 2015-12-20 6: 2 2015-12-25 2014-12-25 2015-12-25 7: 2 2016-02-10 2015-02-10 2016-02-10
result <- DT0[DT0, on = c("patient.id", "hospitalization.date>=start.date",
"hospitalization.date<=end.date"),
.(hospitalizations.last.year = .N), by = .EACHI][]
result
patient.id hospitalization.date hospitalization.date hospitalizations.last.year 1: 1 2012-10-15 2013-10-15 1 2: 1 2014-07-16 2015-07-16 1 3: 1 2015-01-07 2016-01-07 2 4: 2 2013-10-15 2014-10-15 1 5: 2 2014-12-20 2015-12-20 1 6: 2 2014-12-25 2015-12-25 2 7: 2 2015-02-10 2016-02-10 3
除了重命名和重复的列名(保留原样用于比较)。
对于patient.id == 2
,最后一行的结果是3,因为患者在2016-02-10期间自2015-02-10以来第三次住院。
result
是一个新的data.table
对象,占用额外的内存。我尝试使用以下方法更新原始data.table
对象:
# use copy of DT0 which can be safely modified
DT <- copy(DT0)
DT[DT, on = c("patient.id", "hospitalization.date>=start.date",
"hospitalization.date<=end.date"),
hospitalizations.last.year := .N, by = .EACHI]
DT
patient.id hospitalization.date start.date end.date hospitalizations.last.year 1: 1 2013-10-15 2012-10-15 2013-10-15 1 2: 1 2015-07-16 2014-07-16 2015-07-16 2 3: 1 2016-01-07 2015-01-07 2016-01-07 2 4: 2 2014-10-15 2013-10-15 2014-10-15 1 5: 2 2015-12-20 2014-12-20 2015-12-20 3 6: 2 2015-12-25 2014-12-25 2015-12-25 3 7: 2 2016-02-10 2015-02-10 2016-02-10 3
DT
现已更新到位,但第5行和第6行现在显示3次住院而不是1次或2次,分别为。现在看来,每一行都会返回上一期间的住院治疗总数。
non-equi join 条件中的列顺序也很重要,即使在自联接中也是如此:
result <- DT0[DT0, on = c("patient.id", "start.date<=hospitalization.date",
"end.date>=hospitalization.date"),
.(hospitalizations.last.year = .N), by = .EACHI][]
result
我的期望是"start.date<=hospitalization.date"
等同于"hospitalization.date>=start.date"
(请注意<
和>
也已切换)但结果
patient.id start.date end.date hospitalizations.last.year 1: 1 2013-10-15 2013-10-15 1 2: 1 2015-07-16 2015-07-16 2 3: 1 2016-01-07 2016-01-07 1 4: 2 2014-10-15 2014-10-15 1 5: 2 2015-12-20 2015-12-20 3 6: 2 2015-12-25 2015-12-25 2 7: 2 2016-02-10 2016-02-10 1
是不同的。现在似乎正在计算即将到来的住院次数
有趣的是,更新到位现在返回相同的结果(除了一些列名称):
# use copy of DT0 which can be safely modified
DT <- copy(DT0)
DT[DT, on = c("patient.id", "start.date<=hospitalization.date",
"end.date>=hospitalization.date"),
hospitalizations.last.year := .N, by = .EACHI]
DT
patient.id hospitalization.date start.date end.date hospitalizations.last.year 1: 1 2013-10-15 2012-10-15 2013-10-15 1 2: 1 2015-07-16 2014-07-16 2015-07-16 2 3: 1 2016-01-07 2015-01-07 2016-01-07 1 4: 2 2014-10-15 2013-10-15 2014-10-15 1 5: 2 2015-12-20 2014-12-20 2015-12-20 3 6: 2 2015-12-25 2014-12-25 2015-12-25 2 7: 2 2016-02-10 2015-02-10 2016-02-10 1
可能related question导致issue reported on github。
x.
前缀与非等联接有answer by Arun的使用。
答案 0 :(得分:4)
分组by=.EACHI
表示&#34;由每个i&#34;不是&#34;每个x&#34;。
# for readability / my sanity
DT = copy(DT0)
setnames(DT, "hospitalization.date", "h.date")
z = DT[DT, on = .(patient.id, h.date >= start.date, h.date <= end.date),
.(x.h.date, patient.id, i.start.date, i.end.date, g = .GRP, .N)
, by=.EACHI][, utils:::tail.default(.SD, 6)]
x.h.date patient.id i.start.date i.end.date g N
1: 2013-10-15 1 2012-10-15 2013-10-15 1 1 *
2: 2015-07-16 1 2014-07-16 2015-07-16 2 1
3: 2015-07-16 1 2015-01-07 2016-01-07 3 2 *
4: 2016-01-07 1 2015-01-07 2016-01-07 3 2 *
5: 2014-10-15 2 2013-10-15 2014-10-15 4 1 *
6: 2015-12-20 2 2014-12-20 2015-12-20 5 1
7: 2015-12-20 2 2014-12-25 2015-12-25 6 2
8: 2015-12-25 2 2014-12-25 2015-12-25 6 2
9: 2015-12-20 2 2015-02-10 2016-02-10 7 3 *
10: 2015-12-25 2 2015-02-10 2016-02-10 7 3 *
11: 2016-02-10 2 2015-02-10 2016-02-10 7 3 *
对于患者1,这些组是
.(start.date = 2012-10-15, end.date = 2013-10-15)
,数为1 .(start.date = 2014-07-16, end.date = 2015-07-16)
,数为1 .(start.date = 2015-01-07, end.date = 2016-01-07)
,数为2 幸运的是,这个连接中有七个组,原始表中有七个组。
对于更棘手的问题,我会从笔记中借一个例子:
请注意更新联接中的多个匹配。当存在多个匹配项时,更新联接显然仅使用最后一个匹配项。不幸的是,这是默默无闻的。试试:
a = data.table(id = c(1L, 1L, 2L, 3L, NA_integer_), t = c(1L, 2L, 1L, 2L, NA_integer_), x = 11:15) b = data.table(id = 1:2, y = c(11L, 15L)) b[a, on=.(id), x := i.x, verbose = TRUE ][] # Calculated ad hoc index in 0 secs # Starting bmerge ...done in 0.02 secs # Detected that j uses these columns: x,i.x # Assigning to 3 row subset of 2 rows # id y x # 1: 1 11 12 # 2: 2 15 13
在详细介绍的情况下,我们会看到有关“分配到2行的3行子集”的有用信息。
在OP的情况下,遗憾的是verbose=TRUE
不提供此类消息。
DT[DT, on = .(patient.id, h.date >= start.date, h.date <= end.date),
n := .N, by = .EACHI, verbose=TRUE]
# Non-equi join operators detected ...
# forder took ... 0.01 secs
# Generating group lengths ... done in 0 secs
# Generating non-equi group ids ... done in 0 secs
# Found 1 non-equi group(s) ...
# Starting bmerge ...done in 0.02 secs
# Detected that j uses these columns: <none>
# lapply optimization is on, j unchanged as '.N'
# Making each group and running j (GForce FALSE) ...
# memcpy contiguous groups took 0.000s for 7 groups
# eval(j) took 0.000s for 7 calls
# 0.01 secs
但是,我们可以看到每个x
组的最后一行确实包含OP看到的值。我用上面的星号手动标记了这些。或者,您可以使用z[, mrk := replace(rep(0, .N), .N, 1), by=x.h.date]
标记它们。
作为参考,此处的更新加入是......
DT[, n :=
.SD[.SD, on = .(patient.id, h.date >= start.date, h.date <= end.date), .N, by=.EACHI]$N
]
patient.id hospitalization.date start.date end.date h.date n
1: 1 2013-10-15 2012-10-15 2013-10-15 2013-10-15 1
2: 1 2015-07-16 2014-07-16 2015-07-16 2015-07-16 1
3: 1 2016-01-07 2015-01-07 2016-01-07 2016-01-07 2
4: 2 2014-10-15 2013-10-15 2014-10-15 2014-10-15 1
5: 2 2015-12-20 2014-12-20 2015-12-20 2015-12-20 1
6: 2 2015-12-25 2014-12-25 2015-12-25 2015-12-25 2
7: 2 2016-02-10 2015-02-10 2016-02-10 2016-02-10 3
这是处理这种情况的正确/惯用方法,即根据在另一个表中查找x
的每一行并计算结果摘要来向x
添加列:
x[, v := DT2[.SD, on=, j, by=.EACHI]$V1 ]