文本比较中adist函数的问题

时间:2019-07-02 14:20:27

标签: r text comparison text-mining levenshtein-distance

我的adist功能有问题。基本上,我使用的是RDocumentation的示例。

attr(adist(c("kitten", "sitting"), counts = TRUE), "trafos") here

但是,当我尝试跑步时,又增加了一个字

attr(adist(c("kitten", "sitting", "hi"), counts = TRUE), "trafos") 

我得到这些结果:

     [,1]      [,2]      [,3]     
[1,] "MMMMMM"  "SMMMSMI" "SMDDDDI"

[2,] "SMMMSMD" "MMMMMMM" "SDDDMDD"

[3,] "SMIIIID" "SIIIMII" "MMI" 

在第三列第三行中,我正在接受MMI,但由于它是相同的单词“ hi”,所以我无法理解。因此必须是MM。 (匹配,匹配且无插入)

参考:https://www.rdocumentation.org/packages/utils/versions/3.6.0/topics/adist

我正在使用另一个示例:

test <- c('x','hi', 'y','x')

attr(adist(test, y=NULL , counts = TRUE), "trafos")

我正在取得这些结果。但是至少对角线需要为M,因为同一个单词。

     [,1] [,2] [,3] [,4]
[1,] "M"  "SI" "SI" "MI"

[2,] "SD" "MM" "SD" "SD"

[3,] "SD" "SI" "MI" "SI"

[4,] "MI" "SI" "SI" "MI"

我不明白这是怎么回事。

1 个答案:

答案 0 :(得分:6)

正如其他人已经指出的那样,它看起来像个错误。使用https://cran.r-project.org/src/base/R-3/R-3.5.3.tar.gz中的源代码,并查看文件 src / main / agrep.c 中的 429-432 行,其中的代码正在反转缓冲区:< / p>

/* Now reverse the transcript. */
for(k = 0, l = --m; l >= nz; k++, l--)
  buf[k] = buf[l];
buf[++k] = '\0';

了解gdb中发生的情况:

$ R -d gdb
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
...
(gdb) b agrep.c:430
Breakpoint 1 at 0x7222e: file agrep.c, line 430.
(gdb) r
Starting program: /usr/local/lib64/R/bin/exec/R
...
R version 3.5.3 (2019-03-11) -- "Great Truth"
...

然后执行以下R代码:

> attr(adist(c("kitten", "sitting", "hi"), counts = TRUE), "trafos")

Breakpoint 1, adist_full (x=0x555557995a48, y=0x555557995a48, costs=0x5555561567a8, opt_counts=TRUE) at agrep.c:430
430                                 for(k = 0, l = --m; l >= nz; k++, l--)

在中断处继续8,以到达最后一个对角线条目:

(gdb) c 8
Will ignore next 7 crossings of breakpoint 1.  Continuing.

Breakpoint 1, adist_full (x=0x555557995a48, y=0x555557995a48, costs=0x5555561567a8, opt_counts=TRUE) at agrep.c:430
430                                 for(k = 0, l = --m; l >= nz; k++, l--)

在反转之前检查缓冲区:

(gdb) x/6c buf
0x5555566a8da0: 83 'S'  73 'I'  73 'I'  73 'I'  77 'M'  77 'M'

单步执行代码表明buf[0]buf[1]是从缓冲区末尾复制的:

(gdb) n
431                                     buf[k] = buf[l];
(gdb) n
430                                 for(k = 0, l = --m; l >= nz; k++, l--)
(gdb) p k
$1 = 0
(gdb) n
431                                     buf[k] = buf[l];
(gdb) n
430                                 for(k = 0, l = --m; l >= nz; k++, l--)
(gdb) p k
$2 = 1

退出循环k = 2:

(gdb) n
432                                 buf[++k] = '\0';
(gdb) p k
$3 = 2

而++ k是3:

(gdb) n
433                                 COUNTS(i, j, 0) = nins;
(gdb) p k
$4 = 3

检查反向缓冲区,发现buf[2]未设置为NUL:

(gdb) x/6c buf
0x5555566a8da0: 77 'M'  77 'M'  73 'I'  0 '\000'        77 'M'  77 'M'

结果是:

     [,1]      [,2]      [,3]
[1,] "MMMMMM"  "SMMMSMI" "SMDDDDI"
[2,] "SMMMSMD" "MMMMMMM" "SDDDMDD"
[3,] "SMIIIID" "SIIIMII" "MMI"

buf[++k] = '\0'替换buf[k] = '\0'似乎可以将NUL放在正确的位置:

> attr(adist(c("kitten", "sitting", "hi"), counts = TRUE), "trafos")

Breakpoint 1, adist_full (x=0x555557995cb8, y=0x555557995cb8, costs=0x5555561567a8, opt_counts=TRUE) at agrep.c:430
430                                 for(k = 0, l = --m; l >= nz; k++, l--)
(gdb) c 8
Will ignore next 7 crossings of breakpoint 1.  Continuing.

Breakpoint 1, adist_full (x=0x555557995cb8, y=0x555557995cb8, costs=0x5555561567a8, opt_counts=TRUE) at agrep.c:430
430                                 for(k = 0, l = --m; l >= nz; k++, l--)
(gdb) x/6c buf
0x5555566a8da0: 83 'S'  73 'I'  73 'I'  73 'I'  77 'M'  77 'M'
(gdb) n
431                                     buf[k] = buf[l];
(gdb) n
430                                 for(k = 0, l = --m; l >= nz; k++, l--)
(gdb) p k
$1 = 0
(gdb) n
431                                     buf[k] = buf[l];
(gdb) n
430                                 for(k = 0, l = --m; l >= nz; k++, l--)
(gdb) p k
$2 = 1
(gdb) n
432                                 buf[k] = '\0';
(gdb) p k
$3 = 2
(gdb) n
433                                 COUNTS(i, j, 0) = nins;
(gdb) p k
$4 = 2
(gdb) x/6c buf
0x5555566a8da0: 77 'M'  77 'M'  0 '\000'        73 'I'  77 'M'  77 'M'

产生预期的输出:

     [,1]      [,2]      [,3]     
[1,] "MMMMMM"  "SMMMSMI" "SMDDDD" 
[2,] "SMMMSMD" "MMMMMMM" "SDDDMDD"
[3,] "SMIIII"  "SIIIMII" "MM" 

修复后,您的第二个示例导致:

> test <- c('x','hi', 'y','x')
> attr(adist(test, y=NULL , counts = TRUE), "trafos")
     [,1] [,2] [,3] [,4]
[1,] "M"  "SI" "S"  "M"
[2,] "SD" "MM" "SD" "SD"
[3,] "S"  "SI" "M"  "S"
[4,] "M"  "SI" "S"  "M"

结果似乎与ins,sub和del的其他属性一致。

> adist(c('x', 'hi', 'y', 'x'), counts=TRUE)
     [,1] [,2] [,3] [,4]
[1,]    0    2    1    0
[2,]    2    0    2    2
[3,]    1    2    0    1
[4,]    0    2    1    0
attr(,"counts")
, , ins

     [,1] [,2] [,3] [,4]
[1,]    0    1    0    0
[2,]    0    0    0    0
[3,]    0    1    0    0
[4,]    0    1    0    0

, , del

     [,1] [,2] [,3] [,4]
[1,]    0    0    0    0
[2,]    1    0    1    1
[3,]    0    0    0    0
[4,]    0    0    0    0

, , sub

     [,1] [,2] [,3] [,4]
[1,]    0    1    1    0
[2,]    1    0    1    1
[3,]    1    1    0    1
[4,]    0    1    1    0

attr(,"trafos")
     [,1] [,2] [,3] [,4]
[1,] "M"  "SI" "S"  "M"
[2,] "SD" "MM" "SD" "SD"
[3,] "S"  "SI" "M"  "S"
[4,] "M"  "SI" "S"  "M"