读取具有不同列宽但在R中具有固定分隔符的文本文件

时间:2018-03-16 17:41:52

标签: r data.table delimiter space read.table

我有多个.txt文件,如下所示:

header
header
header
header
header
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\AAA AAAAAAAA\AAAAA\BBBB BBBB & BBBBB BBBBB\CAM_07-0008\Farther Downg   Gray Fox                                                                           
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\AAA AAAAAAAA\AAAAA\BBBB BBBB & BBBBB BBBBB\CAM_07-0008\Farther Downg   Direct Register Walk, Gait, Gray Fox, Stop                                         
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\AAA AAAAAAAA\AAAAA\BBBB BBBB & BBBBB BBBBB\CAM_07-0008\Farther Downg   Gray Fox   

最后2列的宽度各不相同,但所有列之间总共有3个空格(在这种情况下,第3列为空)。

我正在使用此代码阅读示例.txt:

read.fwf(filename.txt,skip=5,widths=c(12,16,19,76,83),fill=T,fileEncoding = "UTF-16")

但是这个代码在这个.txt上无法正常工作:

header
header
header
header
header
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\AAA AAAAAAAA\AAAAA AA\BBBB BBBB & BBBBB BBBBB\CAM_07-0008\Farther DowngBBB   Gray Fox                                                                           
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\AAA AAAAAAAA\AAAAA AA\BBBB BBBB & BBBBB BBBBB\CAM_07-0008\Farther DowngBBB   Direct Register Walk, Gait, Gray Fox, Stop                                         
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\AAA AAAAAAAA\AAAAA AA\BBBB BBBB & BBBBB BBBBB\CAM_07-0008\Farther DowngBBB   Gray Fox   

是否有办法使用固定的分隔符(3个空格)读取.txt文件,而不必定义每列的宽度,因为列宽在文件之间不同。

这些文件也有一些编码问题,所以here是我正在使用的示例文件

2 个答案:

答案 0 :(得分:3)

我不知道是否有好的工具可以查找多字符分隔符,而且您不是第一个询问它的人。大多数(包括read.tableread.delimreadr::read_delim)都需要单字节分隔符。

对于大型文件来说,一种方法虽然效率不高,但却是按行加载它们并自己进行拆分。

(底部的消耗性数据。)

x <- readLines(textConnection(file1))
x <- x[x != 'header'] # or x <- x[-(1:5)]

(我猜测它并不总是文字header,所以我假设它是一个固定的数字,或者你可以很容易地知道&#34;知道& #34;哪个是。)

spl <- strsplit(x, '   ')
str(spl)
# List of 3
#  $ : chr [1:31] "01130009.JPG" "JPEG" "" "" ...
#  $ : chr [1:20] "01130009.JPG" "JPEG" "" "" ...
#  $ : chr [1:7] "01130009.JPG" "JPEG" "" "" ...

这似乎没问题,除了在你的例子中,右边有很多空白......

spl[[1]]
#  [1] "01130009.JPG"                                                                
#  [2] "JPEG"                                                                        
#  [3] ""                                                                            
#  [4] ""                                                                            
#  [5] "2/5/2018 3:53:44 PM"                                                         
#  [6] "G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg"
#  [7] "Gray Fox"                                                                    
#  [8] ""                                                                            
#  [9] ""                                                                            
# [10] ""                                                                            
# [11] ""                                                                            
# [12] ""                                                                            
# [13] ""                                                                            
# [14] ""                                                                            
# [15] ""                                                                            
# [16] ""                                                                            
# [17] ""                                                                            
# [18] ""                                                                            
# [19] ""                                                                            
# [20] ""                                                                            
# [21] ""                                                                            
# [22] ""                                                                            
# [23] ""                                                                            
# [24] ""                                                                            
# [25] ""                                                                            
# [26] ""                                                                            
# [27] ""                                                                            
# [28] ""                                                                            
# [29] ""                                                                            
# [30] ""                                                                            
# [31] ""                                                                            

因此,如果您知道有多少列,那么您可以轻松删除其他内容:

spl <- lapply(spl, `[`, 1:7)

然后检查输出:

as.data.frame(do.call(rbind, spl), stringsAsFactors = FALSE)
#             V1   V2 V3 V4                  V5
# 1 01130009.JPG JPEG       2/5/2018 3:53:44 PM
# 2 01130009.JPG JPEG       2/5/2018 3:53:44 PM
# 3 01130009.JPG JPEG       2/5/2018 3:53:44 PM
#                                                                             V6
# 1 G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
# 2 G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
# 3 G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
#                                           V7
# 1                                   Gray Fox
# 2 Direct Register Walk, Gait, Gray Fox, Stop
# 3                                   Gray Fox

这与你的第二个例子同样有效:

x <- readLines(textConnection(file2))
x <- x[x != 'header'] # or x <- x[-(1:5)]
spl <- lapply(strsplit(x, '   '), `[`, 1:7)
as.data.frame(do.call(rbind, spl), stringsAsFactors = FALSE)
#             V1   V2 V3 V4                  V5
# 1 01130009.JPG JPEG       2/5/2018 3:53:44 PM
# 2 01130009.JPG JPEG       2/5/2018 3:53:44 PM
# 3 01130009.JPG JPEG       2/5/2018 3:53:44 PM
#                                                                                   V6
# 1 G:\\AAA AAAAAAAA\\AAAAA AA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther DowngBBB
# 2 G:\\AAA AAAAAAAA\\AAAAA AA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther DowngBBB
# 3 G:\\AAA AAAAAAAA\\AAAAA AA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther DowngBBB
#                                           V7
# 1                                   Gray Fox
# 2 Direct Register Walk, Gait, Gray Fox, Stop
# 3                                   Gray Fox

消耗品数据:

# note: replaced single '\' with double '\\' for R string-handling only
file1 <- 'header
header
header
header
header
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg   Gray Fox                                                                           
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg   Direct Register Walk, Gait, Gray Fox, Stop                                         
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg   Gray Fox   '
file2 <- 'header
header
header
header
header
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\\AAA AAAAAAAA\\AAAAA AA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther DowngBBB   Gray Fox                                                                           
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\\AAA AAAAAAAA\\AAAAA AA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther DowngBBB   Direct Register Walk, Gait, Gray Fox, Stop                                         
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\\AAA AAAAAAAA\\AAAAA AA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther DowngBBB   Gray Fox   '

答案 1 :(得分:2)

可以读取跳过标题行的文件,然后使用gsub函数将3个空格替换为方便的分隔符(此处使用垂直条):

> mytext = "01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg   Gray Fox
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg   Direct Register Walk, Gait, Gray Fox, Stop
01130009.JPG   JPEG         2/5/2018 3:53:44 PM   G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg   Gray Fox"

> ddf = read.table(text=gsub("   ", "|", mytext), header=F, sep="|")
> ddf 
            V1   V2 V3 V4                  V5                                                                           V6
1 01130009.JPG JPEG NA NA 2/5/2018 3:53:44 PM G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
2 01130009.JPG JPEG NA NA 2/5/2018 3:53:44 PM G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
3 01130009.JPG JPEG NA NA 2/5/2018 3:53:44 PM G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
                                          V7
1                                   Gray Fox
2 Direct Register Walk, Gait, Gray Fox, Stop
3                                   Gray Fox

编辑:正如@ r2evans在下面的评论中所建议的那样,必须修剪文本以使用gsub(" *$", "", ...)删除尾随空格。或者,以下函数来自How to trim leading and trailing whitespace in R?

trim.trailing <- function (x) sub("\\s+$", "", x)

对于文本文件,可以使用readLines读取文本文件:

> mytext = readLines(file('testfile.txt')) # read file text
> mytext = mytext[-c(1:5)]           # remove first 5 rows ('header')
> mytext = gsub("\\s+$", "", mytext) # remove trailing spaces
> mytext = gsub("   ", "|", mytext)  # change separator
> ddf = read.table(text=mytext, header=F, sep='|') # read columns from text
> ddf
            V1   V2 V3 V4                  V5                                                                           V6
1 01130009.JPG JPEG NA NA 2/5/2018 3:53:44 PM G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
2 01130009.JPG JPEG NA NA 2/5/2018 3:53:44 PM G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
3 01130009.JPG JPEG NA NA 2/5/2018 3:53:44 PM G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
                                          V7
1                                   Gray Fox
2 Direct Register Walk, Gait, Gray Fox, Stop
3                                   Gray Fox

或者,可以先将它们读取到一个变量的data.frame,然后操纵行以获得所需的结果:

> ddf1 = read.table(file='testfile.txt', sep = '\n', skip=5)
> mytext = gsub("\\s+$", "", unlist(ddf1$V1))
> ddf2 = read.table(text=gsub("   ", "|", mytext), header=F, sep='|')
> ddf2
            V1   V2 V3 V4                  V5                                                                           V6
1 01130009.JPG JPEG NA NA 2/5/2018 3:53:44 PM G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
2 01130009.JPG JPEG NA NA 2/5/2018 3:53:44 PM G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
3 01130009.JPG JPEG NA NA 2/5/2018 3:53:44 PM G:\\AAA AAAAAAAA\\AAAAA\\BBBB BBBB & BBBBB BBBBB\\CAM_07-0008\\Farther Downg
                                          V7
1                                   Gray Fox
2 Direct Register Walk, Gait, Gray Fox, Stop
3                                   Gray Fox