用最近的非NA按组替换缺失值(NA)

时间:2014-04-28 11:42:03

标签: r dplyr

我想用dplyr解决以下问题。优选具有一个窗口功能。 我有一个房屋和购买价格的数据框架。以下是一个例子:

houseID      year    price 
1            1995    NA
1            1996    100
1            1997    NA
1            1998    120
1            1999    NA
2            1995    NA
2            1996    NA
2            1997    NA
2            1998    30
2            1999    NA
3            1995    NA
3            1996    44
3            1997    NA
3            1998    NA
3            1999    NA

我想制作一个这样的数据框:

houseID      year    price 
1            1995    NA
1            1996    100
1            1997    100
1            1998    120
1            1999    120
2            1995    NA
2            1996    NA
2            1997    NA
2            1998    30
2            1999    30
3            1995    NA
3            1996    44
3            1997    44
3            1998    44
3            1999    44

以下是一些格式正确的数据:

# Number of houses
N = 15

# Data frame
df = data.frame(houseID = rep(1:N,each=10), year=1995:2004, price =ifelse(runif(10*N)>0.15, NA,exp(rnorm(10*N))))

有没有一种方法可以做到这一点?

7 个答案:

答案 0 :(得分:59)

tidyr::fill现在让这很简单:

library(dplyr)
library(tidyr)
# or library(tidyverse)

df %>% group_by(houseID) %>% fill(price)
# Source: local data frame [15 x 3]
# Groups: houseID [3]
# 
#    houseID  year price
#      (int) (int) (int)
# 1        1  1995    NA
# 2        1  1996   100
# 3        1  1997   100
# 4        1  1998   120
# 5        1  1999   120
# 6        2  1995    NA
# 7        2  1996    NA
# 8        2  1997    NA
# 9        2  1998    30
# 10       2  1999    30
# 11       3  1995    NA
# 12       3  1996    44
# 13       3  1997    44
# 14       3  1998    44
# 15       3  1999    44

答案 1 :(得分:42)

这些都使用了zoo包中的na.locf。另请注意,na.locf0(也在动物园中定义)与na.locf类似,但默认为na.rm = FALSE并且需要单个向量参数。第一个解决方案中定义的na.locf2也用于其他一些解决方案中。

<强> dplyr

library(dplyr)
library(zoo)

na.locf2 <- function(x) na.locf(x, na.rm = FALSE)
df %>% group_by(houseID) %>% do(na.locf2(.)) %>% ungroup

,并提供:

Source: local data frame [15 x 3]
Groups: houseID

   houseID year price
1        1 1995    NA
2        1 1996   100
3        1 1997   100
4        1 1998   120
5        1 1999   120
6        2 1995    NA
7        2 1996    NA
8        2 1997    NA
9        2 1998    30
10       2 1999    30
11       3 1995    NA
12       3 1996    44
13       3 1997    44
14       3 1998    44
15       3 1999    44

其中一个变体是:

df %>% group_by(houseID) %>% mutate(price = na.locf0(price)) %>% ungroup

下面的其他解决方案给出了非常相似的输出,因此我们不会重复它,除非格式大不相同。

另一种可能性是将by解决方案(如下所示)与dplyr结合起来:

df %>% by(df$houseID, na.locf2) %>% bind_rows

library(zoo)

do.call(rbind, by(df, df$houseID, na.locf2))

<强> AVE

library(zoo)

transform(df, price = ave(price, houseID, FUN = na.locf0))

<强> data.table

library(data.table)
library(zoo)

data.table(df)[, na.locf2(.SD), by = houseID]

动物园此解决方案仅使用动物园。它返回一个宽而不是长的结果:

library(zoo)

z <- read.zoo(df, index = 2, split = 1, FUN = identity)
na.locf2(z)

,并提供:

       1  2  3
1995  NA NA NA
1996 100 NA 44
1997 100 NA 44
1998 120 30 44
1999 120 30 44

此解决方案可以与dplyr结合使用:

library(dplyr)
library(zoo)

df %>% read.zoo(index = 2, split = 1, FUN = identity) %>% na.locf2

<强>输入

以下是上述示例的输入:

df <- structure(list(houseID = c(1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 
  2L, 3L, 3L, 3L, 3L, 3L), year = c(1995L, 1996L, 1997L, 1998L, 
  1999L, 1995L, 1996L, 1997L, 1998L, 1999L, 1995L, 1996L, 1997L, 
  1998L, 1999L), price = c(NA, 100L, NA, 120L, NA, NA, NA, NA, 
  30L, NA, NA, 44L, NA, NA, NA)), .Names = c("houseID", "year", 
  "price"), class = "data.frame", row.names = c(NA, -15L))

已修订重新安排并添加更多解决方案。修订了dplyr / zoo解决方案以符合最新的更改dplyr。从所有解决方案中应用固定和分解na.locf2

答案 2 :(得分:13)

您可以执行滚动自加入,由data.table支持:

require(data.table)
setDT(df)   ## change it to data.table in place
setkey(df, houseID, year)     ## needed for fast join
df.woNA <- df[!is.na(price)]  ## version without the NA rows

# rolling self-join will return what you want
df.woNA[df, roll=TRUE]  ## will match previous year if year not found

答案 3 :(得分:9)

Pure dplyr解决方案(没有动物园)。

df %>% 
 group_by(houseID) %>%
 mutate(price_change = cumsum(0 + !is.na(price))) %>%
 group_by(price_change, add = TRUE) %>%
 mutate(price_filled = nth(price, 1)) %>%
 ungroup() %>%
 select(-price_change) -> df2

示例解决方案的有趣部分是在df2的末尾。

> tail(df2, 20)
Source: local data frame [20 x 4]

    houseID year     price price_filled
 1       14 1995        NA           NA
 2       14 1996        NA           NA
 3       14 1997        NA           NA
 4       14 1998        NA           NA
 5       14 1999 0.8374778    0.8374778
 6       14 2000        NA    0.8374778
 7       14 2001        NA    0.8374778
 8       14 2002        NA    0.8374778
 9       14 2003 2.1918880    2.1918880
10       14 2004        NA    2.1918880
11       15 1995        NA           NA
12       15 1996 0.3982450    0.3982450
13       15 1997        NA    0.3982450
14       15 1998 1.7727000    1.7727000
15       15 1999        NA    1.7727000
16       15 2000        NA    1.7727000
17       15 2001        NA    1.7727000
18       15 2002 7.8636329    7.8636329
19       15 2003        NA    7.8636329
20       15 2004        NA    7.8636329

答案 4 :(得分:2)

没有dplyr

  prices$price <-unlist(lapply(split(prices$price,prices$houseID),
function(x) zoo::na.locf(x,na.rm=FALSE)))

prices
   houseID year price
1        1 1995    NA
2        1 1996   100
3        1 1997   100
4        1 1998   120
5        1 1999   120
6        2 1995    NA
7        2 1996    NA
8        2 1997    NA
9        2 1998    30
10       2 1999    30
11       3 1995    NA
12       3 1996    44
13       3 1997    44
14       3 1998    44
15       3 1999    44

答案 5 :(得分:2)

dplyrimputeTS的组合。

library(dplyr)
library(imputeTS)
df %>% group_by(houseID) %>% 
mutate(price = na.locf(price, na.remaining="keep"))  

您还可以用na.locf中的更高级的丢失数据替换(输入)功能替换imputeTS。例如na.interpolationna.kalman。为此,只需将na.locf替换为您喜欢的函数的名称即可。

答案 6 :(得分:1)

data.table v1.12.4起,该软件包具有nafill()功能,类似于tidyr::fill()zoo::na.locf(),您可以执行以下操作:

require(data.table)
setDT(df)

df[ , price := nafill(price, type = 'locf'), houseID ]

还有setnafill(),尽管不允许 group by ,但多列。

setnafill(df, type = 'locf', cols = 'price')

数据来自@G。格洛腾迪克的答案:

df = data.frame(houseID = c(1L, 1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 
                            2L, 3L, 3L, 3L, 3L, 3L),
                year = c(1995L, 1996L, 1997L, 1998L, 1999L, 1995L, 1996L,
                         1997L, 1998L, 1999L, 1995L, 1996L, 1997L, 1998L, 1999L),
                price = c(NA, 100L, NA, 120L, NA, NA, NA, NA, 30L, NA, NA, 44L,
                          NA, NA, NA))