Question

免责声明：我对SparkR没什么经验

采用以下数据框：

ID          Date1       Date2
58844880    04/11/16    NaN
59745846    04/12/16    04/14/16
59743311    04/13/16    NaN
59745848    04/14/16    04/11/16
59598413    NaN         NaN
59745921    04/14/16    04/14/16
59561199    04/15/16    04/15/16
NaN         04/16/16    04/16/16
59561198    NaN         04/17/16

我想在日期2列中抓取NaN的行。

在R中，我会使用new_DF <- DF[is.na(DF$Var),]，但我不确定在SparkR中做什么

Answer 1

可以使用条件为filter的{{1}}：

isNull

Answer 2

以下是使用sparklyr

的一个选项

library(sparklyr)
library(dplyr)

con <- spark_connect(master = "local")
DF1 = copy_to(con, DF)


DF1 %>%
   mutate_at(vars(matches("Date")), 
          funs(to_date(from_unixtime(unix_timestamp(., "MM/dd/yy"))))) %>%
   filter(is.na(Date2)) %>%
   collect()
# A tibble: 3 x 3
#        ID Date1      Date2     
#     <dbl> <date>     <date>    
#1 58844880 2016-04-10 NA        
#2 59743311 2016-04-12 NA        
#3 59598413 NA         NA      

spark_disconnect(con)

数据

DF <- structure(list(ID = c(58844880, 59745846, 59743311, 59745848, 
 59598413, 59745921, 59561199, NaN, 59561198), Date1 = c("04/11/16", 
 "04/12/16", "04/13/16", "04/14/16", "NaN", "04/14/16", "04/15/16", 
 "04/16/16", "NaN"), Date2 = c("NaN", "04/14/16", "NaN", "04/11/16", 
 "NaN", "04/14/16", "04/15/16", "04/16/16", "04/17/16")), .Names = c("ID", 
 "Date1", "Date2"), class = "data.frame", row.names = c(NA, -9L
 ))

SparkR：如何提取特定列

2 个答案:

数据