我有一个模糊的TSV，我正在尝试阅读，显然它以标识符开头，并嵌入了一些NUL值（似乎它是每个真正字符后的一个NUL）。这些是文件的前100个字节（用十六进制编辑器缩短）：test_file.txt（我必须将其重命名为txt才能上传它，但它是一个tsv文件）。
不幸的是，我无法使用基本函数读取它，也无法使用readr或data.table读取它。
这是代表：
file <- 'test_file.txt'
# read.tsv is not able to read the file since there are embedded NULs
tmp <- read.table(file, header = T, nrows = 2)
#> Warning in read.table(file, header = T, nrows = 2): line 1 appears to
#> contain embedded nulls
#> Warning in read.table(file, header = T, nrows = 2): line 2 appears to
#> contain embedded nulls
#> Warning in read.table(file, header = T, nrows = 2): line 3 appears to
#> contain embedded nulls
#> Warning in scan(file = file, what = what, sep = sep, quote = quote, dec =
#> dec, : embedded nul(s) found in input
# unfortunately the skipNul argument also doesn't work
tmp <- read.table(file, header = T, nrows = 2, skipNul = T)
#> Error in read.table(file, header = T, nrows = 2, skipNul = T): more columns than column names
# read_tsv from readr is also not able to read the file (probably since it stops each line after a NUL)
tmp <- readr::read_tsv(file, n_max = 2)
#> Warning: Duplicated column names deduplicated: '' => '_1' [3], '' =>
#> '_2' [4], '' => '_3' [5], '' => '_4' [6], '' => '_5' [7], '' => '_6' [8],
#> '' => '_7' [9], '' => '_8' [10], '' => '_9' [11], '' => '_10' [12], '' =>
#> '_11' [13]
#> Parsed with column specification:
#> cols(
#> y = col_character(),
#> col_character(),
#> `_1` = col_character(),
#> `_2` = col_character(),
#> `_3` = col_character(),
#> `_4` = col_character(),
#> `_5` = col_character(),
#> `_6` = col_character(),
#> `_7` = col_character(),
#> `_8` = col_character(),
#> `_9` = col_character(),
#> `_10` = col_character(),
#> `_11` = col_character()
#> )
#> Error in read_tokens_(data, tokenizer, col_specs, col_names, locale_, : Column 2 must be named
# fread from data.table is also not able to read the file (although it is the first function that more clearly shows the problem)
tmp <- data.table::fread(file, nrows = 2)
#> Error in data.table::fread(file, nrows = 2): embedded nul in string: 'ÿþy\0e\0a\0r\0'
# read lines reads the first actual character 'y' and the file identifier characters that seem to parse as 'ÿþ' in UTF-8
readLines(file, n = 1)
#> Warning in readLines(file, n = 1): line 1 appears to contain an embedded
#> nul
#> [1] "ÿþy"
# the problem is in the hidden NUL characters as the following command shows
readLines(file, n = 1, skipNul = T)
#> [1] "ÿþyear\tmonth\tday\tDateTime\tAreaTypeCode\tAreaName\tMapCode\tPowerSystemResourceName\tProductionTypeName\tActualGenerationOutput\tActualConsumption\tInstalledGenCapacity\tSubmissionTS"
是否有可以让我阅读此文件的解决方法？最好不要通过基本功能，因为它们非常慢，我必须读取超过300 MB的多个文件（> 20）。
答案 0 :(得分：1)
目前的变通方法在Removing "NUL" characters (within R)
中有所描述这个答案很大程度上依赖于那一个。我添加了一些注释，修改了示例以使用头字节，并添加了fread（data.table）和read_tsv（readr）的用法，以建立到数据帧的最终链接。
file <- 'test_file.txt'
# read the file as raw and skip the first two header bytes
data_raw <- readBin(file, raw(), file.info(file)$size)[3:file.info(file)$size]
# replace the NUL values by an uncommon UTF-8 character so that we can
# later filter this one out. Please check out this list for more uncommon
# characters: http://www.fileformat.info/info/charset/UTF-8/list.htm
data_raw[data_raw == as.raw(0)] <- as.raw(1)
# convert to a string and remove the replaced characters (raw(1) in our case)
data_string <- gsub("\001", "", rawToChar(data_raw), fixed = TRUE)
# convert the resulting string to a data frame by a function to your liking
data_tmp1 <- data.table::fread(data_string, header = T) # quickest
data_tmp2 <- readr::read_tsv(data_string) # slower and is not working well with the UTF-8 characters
data_tmp3 <- read.table(data_string) # crashed R for my files (probably due to size)
此方法已在相对较大的文件（350 MB）上进行了测试。第一步（readBin）是最慢的，但对于此文件大小只需要约30秒。速度可能还取决于您的硬盘驱动器。
对于非常大的文件，也可能存在内存问题。与fread相反，在进行任何处理之前，所有内容都将被读入内存。