Question

我的表格中有一些数据：

          V1   V2  V3         V4  V5         V6   V7  V8  V9 V10 V11
 1. 14342667 4336 1.5 2015-10-03 \\N 2015-10-03  804 2.0 2.0   0 3.0
 2. 14342667 4336 0.8 2015-06-13 \\N 2015-06-11 2912 2.0 2.0   0 \\N
 3. 14342667 4336 0.5 2016-01-02 \\N 2015-12-27 1618 0.0 0.0   1 \\N
 4. 14342667 4336 0.7 2015-08-22 \\N 2015-08-22 1780 2.0 2.0   0 \\N
 5. 14342667 4336 0.9 2015-02-21 1.2 2015-02-17 1548 0.0 0.0   1 \\N
 6. 14342667 4336 1.0 2015-08-08 \\N 2015-08-06 1538 2.0 2.0   0 2.25
 7. 14342667 4336 0.9 2015-03-28 \\N 2015-03-24 2129 7.0 7.0   0 \\N
 8. 14342667 4336 0.8 2015-04-11 \\N 2015-04-11 2316 1.0 2.0   0 \\N

我想使用R和data.table将\\N的实例替换为NA。当我将数据扩展到HIVE时，我得到了这些空白。

我尝试使用CAR包这样的data1 <- data1[, lapply(.SD, recode, '"\\N"=NA')]和data1 <- data1[, lapply(.SD, recode, '"\N"=NA')]这样的东西适用于其他替换但在当前情况下失败但错误：

FUN错误（X [[1L]]，...）：在重新编码术语中：＆＃34; \ N＆＃34; = NA 消息：错误：＆＃39; \ N＆＃39;是一个无法识别的转义字符串开始＆＃34;＆＃34; \ N＆＃34;

我也阅读了像x[x=="\\N"] <- NA这样的解决方案，但我无法成功使用data.table。我还研究了HIVE表格方面的解决方案，但显然regexp_replace仅适用于一列而不适用于所有列中的所有实例。

Answer 1

我们可以使用grep找出具有此模式的列，然后在这些列上使用as.numeric

library(data.table)
nm1 <- names(df1)[colSums(sapply(df1, grepl, pattern = "\\\\"))!=0]
setDT(df1)[,(nm1):= lapply(.SD, as.numeric) , .SDcols= nm1]
df1
#         V1   V2  V3         V4  V5         V6   V7 V8 V9 V10  V11
#1: 14342667 4336 1.5 2015-10-03  NA 2015-10-03  804  2  2   0 3.00
#2: 14342667 4336 0.8 2015-06-13  NA 2015-06-11 2912  2  2   0   NA
#3: 14342667 4336 0.5 2016-01-02  NA 2015-12-27 1618  0  0   1   NA
#4: 14342667 4336 0.7 2015-08-22  NA 2015-08-22 1780  2  2   0   NA
#5: 14342667 4336 0.9 2015-02-21 1.2 2015-02-17 1548  0  0   1   NA
#6: 14342667 4336 1.0 2015-08-08  NA 2015-08-06 1538  2  2   0 2.25
#7: 14342667 4336 0.9 2015-03-28  NA 2015-03-24 2129  7  7   0   NA
#8: 14342667 4336 0.8 2015-04-11  NA 2015-04-11 2316  1  2   0   NA

数据

df1 <- structure(list(V1 = c(14342667L, 14342667L, 14342667L, 14342667L, 
14342667L, 14342667L, 14342667L, 14342667L), V2 = c(4336L, 4336L, 
4336L, 4336L, 4336L, 4336L, 4336L, 4336L), V3 = c(1.5, 0.8, 0.5, 
0.7, 0.9, 1, 0.9, 0.8), V4 = c("2015-10-03", "2015-06-13", "2016-01-02", 
"2015-08-22", "2015-02-21", "2015-08-08", "2015-03-28", "2015-04-11"
), V5 = c("\\\\N", "\\\\N", "\\\\N", "\\\\N", "1.2", "\\\\N", 
"\\\\N", "\\\\N"), V6 = c("2015-10-03", "2015-06-11", "2015-12-27", 
"2015-08-22", "2015-02-17", "2015-08-06", "2015-03-24", "2015-04-11"
), V7 = c(804L, 2912L, 1618L, 1780L, 1548L, 1538L, 2129L, 2316L
), V8 = c(2, 2, 0, 2, 0, 2, 7, 1), V9 = c(2, 2, 0, 2, 0, 2, 7, 
2), V10 = c(0L, 0L, 1L, 0L, 1L, 0L, 0L, 0L), V11 = c("3.0", "\\\\N", 
"\\\\N", "\\\\N", "\\\\N", "2.25", "\\\\N", "\\\\N")), .Names = c("V1", 
"V2", "V3", "V4", "V5", "V6", "V7", "V8", "V9", "V10", "V11"), 
 class = "data.frame", row.names = c("1.", 
"2.", "3.", "4.", "5.", "6.", "7.", "8."))

Answer 2

在@ akrun的建议中，这是另一种简单的方法：

ccols = which(sapply(DF, class) == "character")
DF[ccols] <- lapply(DF[ccols], type.convert, na.strings="\\\\N")

使用str(DF)查看列应该是什么（数字或整数）。

使用data.table，这看起来像

library(data.table)
setDT(DF)
ccols = which(sapply(DF, class) == "character")
DF[, (ccols) := lapply(.SD, type.convert, na.strings="\\\\N"), .SDcols=ccols]

Answer 3

NULL在HDFS中的Hive中看起来像//N所以如果寻找HIVE端解决方案，也可以使用函数nvl()进行替换。

使用R＆amp; N替换为NA的\\ N data.table或HIVE

3 个答案:

数据