R如何处理凌乱的数据格式?

时间:2018-08-26 06:21:12

标签: r dataframe

有时我会看到以this question格式格式化的堆栈溢出问题中发布的数据。这不是第一次,所以我决定问一个问题,并以使发布的数据可口的方式回答该问题。

我将在此处发布数据集示例,以防万一问题被删除。

+------------+------+------+----------+--------------------------+
|    Date    | Emp1 | Case | Priority | PriorityCountinLast7days |
+------------+------+------+----------+--------------------------+
| 2018-06-01 | A    | A1   |        0 |                        0 |
| 2018-06-03 | A    | A2   |        0 |                        1 |
| 2018-06-03 | A    | A3   |        0 |                        2 |
| 2018-06-03 | A    | A4   |        1 |                        1 |
| 2018-06-03 | A    | A5   |        2 |                        1 |
| 2018-06-04 | A    | A6   |        0 |                        3 |
| 2018-06-01 | B    | B1   |        0 |                        1 |
| 2018-06-02 | B    | B2   |        0 |                        2 |
| 2018-06-03 | B    | B3   |        0 |                        3 |
+------------+------+------+----------+--------------------------+

如您所见,这不是发布数据的正确方法。正如a user在评论中写道,

  

以某种方式格式化数据必须花费一些时间   在这里显示。不幸的是,这不是我们的好格式   复制并粘贴。

我相信这说明了一切。提问者的意图很强,花了一些时间和精力才能变得更好,但结果并不理想。

如果有的话,R代码可以做什么使该表可用?这会带来很多麻烦吗?

6 个答案:

答案 0 :(得分:25)

使用data.table::fread

x = '
+------------+------+------+----------+--------------------------+
|    Date    | Emp1 | Case | Priority | PriorityCountinLast7days |
+------------+------+------+----------+--------------------------+
| 2018-06-01 | A    | A1   |        0 |                        0 |
| 2018-06-03 | A    | A2   |        0 |                        1 |
| 2018-06-03 | A    | A3   |        0 |                        2 |
| 2018-06-03 | A    | A4   |        1 |                        1 |
| 2018-06-03 | A    | A5   |        2 |                        1 |
| 2018-06-04 | A    | A6   |        0 |                        3 |
| 2018-06-01 | B    | B1   |        0 |                        1 |
| 2018-06-02 | B    | B2   |        0 |                        2 |
| 2018-06-03 | B    | B3   |        0 |                        3 |
+------------+------+------+----------+--------------------------+
'

fread(gsub('\\+.+\\n' ,'', x, perl = T), drop=c(1,7))

#          Date Emp1 Case Priority PriorityCountinLast7days
# 1: 2018-06-01    A   A1        0                        0
# 2: 2018-06-03    A   A2        0                        1
# 3: 2018-06-03    A   A3        0                        2
# 4: 2018-06-03    A   A4        1                        1
# 5: 2018-06-03    A   A5        2                        1
# 6: 2018-06-04    A   A6        0                        3
# 7: 2018-06-01    B   B1        0                        1
# 8: 2018-06-02    B   B2        0                        2
# 9: 2018-06-03    B   B3        0                        3

gsub部分删除水平线。 drop删除了由行尾定界符引起的多余列。

答案 1 :(得分:18)

这个问题的简短答案是肯定的,R代码可以解决麻烦,而不是,它不会带来太多麻烦。

将表复制并粘贴到R会话后的第一步是使用read.table设置headersepcomment.char和{{1 }}参数。

贷方使我想起了论点strip.whitecomment.char到@nicola,以及他的评论。

strip.white

但是您可以看到结果存在一些问题。

dat <- read.table(text = "
+------------+------+------+----------+--------------------------+
|    Date    | Emp1 | Case | Priority | PriorityCountinLast7days |
+------------+------+------+----------+--------------------------+
| 2018-06-01 | A    | A1   |        0 |                        0 |
| 2018-06-03 | A    | A2   |        0 |                        1 |
| 2018-06-03 | A    | A3   |        0 |                        2 |
| 2018-06-03 | A    | A4   |        1 |                        1 |
| 2018-06-03 | A    | A5   |        2 |                        1 |
| 2018-06-04 | A    | A6   |        0 |                        3 |
| 2018-06-01 | B    | B1   |        0 |                        1 |
| 2018-06-02 | B    | B2   |        0 |                        2 |
| 2018-06-03 | B    | B3   |        0 |                        3 |
+------------+------+------+----------+--------------------------+
", header = TRUE, sep = "|", comment.char = "+", strip.white = TRUE)

要使分隔符在每个数据行的开头和结尾处都使R相信这些分隔符会标记额外的列,这不是原始问题的OP的含义。

所以第二步是只保留 real 列。我将按其编号对列进行分组,这很容易做到,它们通常是第一列和最后一列。

dat
   X       Date Emp1 Case Priority PriorityCountinLast7days X.1
1 NA 2018-06-01    A   A1        0                        0  NA
2 NA 2018-06-03    A   A2        0                        1  NA
3 NA 2018-06-03    A   A3        0                        2  NA
4 NA 2018-06-03    A   A4        1                        1  NA
5 NA 2018-06-03    A   A5        2                        1  NA
6 NA 2018-06-04    A   A6        0                        3  NA
7 NA 2018-06-01    B   B1        0                        1  NA
8 NA 2018-06-02    B   B2        0                        2  NA
9 NA 2018-06-03    B   B3        0                        3  NA

不太难,要好得多。
在这种情况下,仍然存在将列dat <- dat[-c(1, ncol(dat))] dat Date Emp1 Case Priority PriorityCountinLast7days 1 2018-06-01 A A1 0 0 2 2018-06-03 A A2 0 1 3 2018-06-03 A A3 0 2 4 2018-06-03 A A4 1 1 5 2018-06-03 A A5 2 1 6 2018-06-04 A A6 0 3 7 2018-06-01 B B1 0 1 8 2018-06-02 B B2 0 2 9 2018-06-03 B B3 0 3 强制转换为类Date的问题。

Date

结果令人满意。

dat$Date <- as.Date(dat$Date)

请注意,我尚未设置或多或少的标准参数str(dat) 'data.frame': 9 obs. of 5 variables: $ Date : Date, format: "2018-06-01" "2018-06-03" ... $ Emp1 : Factor w/ 2 levels "A","B": 1 1 1 1 1 1 2 2 2 $ Case : Factor w/ 9 levels "A1","A2","A3",..: 1 2 3 4 5 6 7 8 9 $ Priority : int 0 0 0 1 2 0 0 0 0 $ PriorityCountinLast7days: int 0 1 2 1 1 3 1 2 3 。如果需要,应在运行stringsAsFactors = FALSE时执行此操作。

整个过程仅需3行基本R代码。

最后,最终结果采用read.table格式,就像它应该放在首位一样。

dput

答案 2 :(得分:5)

md_table <- scan(text = "
+------------+------+------+----------+--------------------------+
|    Date    | Emp1 | Case | Priority | PriorityCountinLast7days |
+------------+------+------+----------+--------------------------+
| 2018-06-01 | A    | A1   |        0 |                        0 |
| 2018-06-03 | A    | A2   |        0 |                        1 |
| 2018-06-03 | A    | A3   |        0 |                        2 |
| 2018-06-03 | A    | A4   |        1 |                        1 |
| 2018-06-03 | A    | A5   |        2 |                        1 |
| 2018-06-04 | A    | A6   |        0 |                        3 |
| 2018-06-01 | B    | B1   |        0 |                        1 |
| 2018-06-02 | B    | B2   |        0 |                        2 |
| 2018-06-03 | B    | B3   |        0 |                        3 |
+------------+------+------+----------+--------------------------+",
what = "", sep = "", comment.char = "+", quiet = TRUE)

## it is clear that there are 5 columns
mat <- matrix(md_table[md_table != "|"], ncol = 5, byrow = TRUE)
#      [,1]         [,2]   [,3]   [,4]       [,5]                      
# [1,] "Date"       "Emp1" "Case" "Priority" "PriorityCountinLast7days"
# [2,] "2018-06-01" "A"    "A1"   "0"        "0"                       
# [3,] "2018-06-03" "A"    "A2"   "0"        "1"                       
# [4,] "2018-06-03" "A"    "A3"   "0"        "2"                       
# [5,] "2018-06-03" "A"    "A4"   "1"        "1"                       
# [6,] "2018-06-03" "A"    "A5"   "2"        "1"                       
# [7,] "2018-06-04" "A"    "A6"   "0"        "3"                       
# [8,] "2018-06-01" "B"    "B1"   "0"        "1"                       
# [9,] "2018-06-02" "B"    "B2"   "0"        "2"                       
#[10,] "2018-06-03" "B"    "B3"   "0"        "3"

## a data frame with all character columns
dat <- setNames(data.frame(mat[-1, ], stringsAsFactors = FALSE), mat[1, ])
#        Date Emp1 Case Priority PriorityCountinLast7days
#1 2018-06-01    A   A1        0                        0
#2 2018-06-03    A   A2        0                        1
#3 2018-06-03    A   A3        0                        2
#4 2018-06-03    A   A4        1                        1
#5 2018-06-03    A   A5        2                        1
#6 2018-06-04    A   A6        0                        3
#7 2018-06-01    B   B1        0                        1
#8 2018-06-02    B   B2        0                        2
#9 2018-06-03    B   B3        0                        3

## or maybe just use `type.convert` on some columns?
dat[] <- lapply(dat, type.convert)

答案 3 :(得分:4)

问题不在于需要多少行代码,两到五行,相差不大。问题是,它是否还能超出您在此处发布的示例的范围?

我没有在野外遇到过这种事情,但是我尝试构建另一个我认为可能存在的示例。


此后,我又遇到了一些案例,并将它们添加到测试套件中。

我还提供了一个使用box-drawing characters绘制的表格。这些天你不会碰到太多,但是为了完整起见,它在这里。

x1 <- "
+------------+------+------+----------+--------------------------+
|    Date    | Emp1 | Case | Priority | PriorityCountinLast7days |
+------------+------+------+----------+--------------------------+
| 2018-06-01 | A    | A1   |        0 |                        0 |
| 2018-06-03 | A    | A2   |        0 |                        1 |
| 2018-06-02 | B    | B2   |        0 |                        2 |
| 2018-06-03 | B    | B3   |        0 |                        3 |
+------------+------+------+----------+--------------------------+
"

x2 <- "
––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
    Date    | Emp1 | Case | Priority | PriorityCountinLast7days 
––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
 2018-06-01 | A    | A|1  |        0 |                        0 
 2018-06-03 | A    | A|2  |        0 |                        1 
 2018-06-02 | B    | B|2  |        0 |                        2 
 2018-06-03 | B    | B|3  |        0 |                        3 
––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
"

x3 <- "
 Maths | English | Science | History | Class

  0.1  |  0.2    |  0.3    |  0.2    |  Y2

  0.9  |  0.5    |  0.7    |  0.4    |  Y1

  0.2  |  0.4    |  0.6    |  0.2    |  Y2

  0.9  |  0.5    |  0.2    |  0.7    |  Y1
"

x4 <- "
       Season   |   Team  | W | AHWO
-------------------------------------
1  |  2017/2018 |  TeamA  | 2 | 1.75
2  |  2017/2018 |  TeamB  | 1 | 1.85
3  |  2017/2018 |  TeamC  | 1 | 1.70
4  |  2016/2017 |  TeamA  | 1 | 1.49
5  |  2016/2017 |  TeamB  | 3 | 1.51
6  |  2016/2017 |  TeamC  | 2 | N/A
"

x5 <- "
    A   B   C
  ┌───┬───┬───┐
A │ 5 │ 1 │ 4 │
  ├───┼───┼───┤
B │ 2 │ 5 │ 3 │
  ├───┼───┼───┤
C │ 3 │ 4 │ 4 │
  └───┴───┴───┘
"

x6 <- "
------------------------------------------------------------
|date              |Material          |Description         |
|----------------------------------------------------------|
|10/04/2013        |WM.5597394        |PNEUMATIC           |
|11/07/2013        |GB.D040790        |RING                |
------------------------------------------------------------
------------------------------------------------------------
|date              |Material          |Description         |
|----------------------------------------------------------|
|08/06/2013        |WM.4M01004A05     |TOUCHEUR            |
|08/06/2013        |WM.4M010108-1     |LEVER               |
------------------------------------------------------------
"

我去参加一项功能

f <- function(x=x6, header=TRUE, rem.dup.header=header, 
  na.strings=c("NA", "N/A"), stringsAsFactors=FALSE, ...) {

    # read each row as a character string
    x <- scan(text=x, what="character", sep="\n", quiet=TRUE)

    # keep only lines containing alphanumerics
    x <- x[grep("[[:alnum:]]", x)]

    # remove vertical bars with trailing or leading space
    x <- gsub("\\|? | \\|?", " ", x)

    # remove vertical bars at beginning and end of string
    x <- gsub("\\|?$|^\\|?", "", x)

    # remove vertical box-drawing characters
    x <- gsub("\U2502|\U2503|\U2505|\U2507|\U250A|\U250B", " ", x)

    if (rem.dup.header) {
        dup.header <- x == x[1]
        dup.header[1] <- FALSE
        x <- x[!dup.header]
    }

    # read the result as a table
    read.table(text=paste(x, collapse="\n"), header=header, 
      na.strings=na.strings, stringsAsFactors=stringsAsFactors)    
}


lapply(c(x1, x2, x3, x4, x5, x6), f)

输出

[[1]]
        Date Emp1 Case Priority PriorityCountinLast7days
1 2018-06-01    A   A1        0                        0
2 2018-06-03    A   A2        0                        1
3 2018-06-02    B   B2        0                        2
4 2018-06-03    B   B3        0                        3

[[2]]
        Date Emp1 Case Priority PriorityCountinLast7days
1 2018-06-01    A  A|1        0                        0
2 2018-06-03    A  A|2        0                        1
3 2018-06-02    B  B|2        0                        2
4 2018-06-03    B  B|3        0                        3

[[3]]
  Maths English Science History Class
1   0.1     0.2     0.3     0.2    Y2
2   0.9     0.5     0.7     0.4    Y1
3   0.2     0.4     0.6     0.2    Y2
4   0.9     0.5     0.2     0.7    Y1

[[4]]
     Season  Team W AHWO
1 2017/2018 TeamA 2 1.75
2 2017/2018 TeamB 1 1.85
3 2017/2018 TeamC 1 1.70
4 2016/2017 TeamA 1 1.49
5 2016/2017 TeamB 3 1.51
6 2016/2017 TeamC 2   NA

[[5]]
  A B C
A 5 1 4
B 2 5 3
C 3 4 4

[[6]]
        date      Material Description
1 10/04/2013    WM.5597394   PNEUMATIC
2 11/07/2013    GB.D040790        RING
3 08/06/2013 WM.4M01004A05    TOUCHEUR
4 08/06/2013 WM.4M010108-1       LEVER

x3来自here(必须查看编辑历史记录)。
x4来自here
x6来自here

答案 4 :(得分:2)

好吧,关于这个特定的数据集,我在RStudio中使用了导入功能,但是我事先又采取了另一步骤。

  1. 将数据集复制到记事本文件中。
  2. 将所有|个字符替换为,
  3. Import使用read.csv到RStudio的记事本文件(使用,分隔的列)。

但是,如果您要使用R一步来完全理解它,那么我不知道。

答案 5 :(得分:-3)

按照建议,您可以使用 dput 将数据框的内容保存到文件中,在文本编辑器中打开文件并粘贴其内容。仅限前10行的mtcar数据集示例:

dput(mtcars  %>% head(10), file = 'reproducible.txt')

reproducible.txt的内容可用于制作数据框/ tibble,如下所示。在这种情况下,数据的格式是机器可读的,但乍一看(不粘贴到R中)很难被人理解。

df <- structure(list(mpg = c(21, 21, 22.8, 21.4, 18.7, 18.1, 14.3,
24.4, 22.8, 19.2), cyl = c(6, 6, 4, 6, 8, 6, 8, 4, 4, 6), disp = c(160,
160, 108, 258, 360, 225, 360, 146.7, 140.8, 167.6), hp = c(110,
110, 93, 110, 175, 105, 245, 62, 95, 123), drat = c(3.9, 3.9,
3.85, 3.08, 3.15, 2.76, 3.21, 3.69, 3.92, 3.92), wt = c(2.62,
2.875, 2.32, 3.215, 3.44, 3.46, 3.57, 3.19, 3.15, 3.44), qsec = c(16.46,
17.02, 18.61, 19.44, 17.02, 20.22, 15.84, 20, 22.9, 18.3), vs = c(0,
0, 1, 1, 0, 1, 0, 1, 1, 1), am = c(1, 1, 1, 0, 0, 0, 0, 0, 0,
0), gear = c(4, 4, 4, 3, 3, 3, 3, 4, 4, 4), carb = c(4, 4, 1,
1, 2, 1, 4, 2, 2, 4)), .Names = c("mpg", "cyl", "disp", "hp",
"drat", "wt", "qsec", "vs", "am", "gear", "carb"), row.names = c("Mazda RX4",
"Mazda RX4 Wag", "Datsun 710", "Hornet 4 Drive", "Hornet Sportabout",
"Valiant", "Duster 360", "Merc 240D", "Merc 230", "Merc 280"), class = "data.frame")