Question

我有一个制表符分隔的文件，其中第12列（从1开始）包含几个以逗号分隔的标识符。但是，它们中的一些在同一行中可能会出现不止一次：

GO:0042302, GO:0042302, GO:0042302
GO:0004386,GO:0005524,GO:0006281, GO:0004386,GO:0005524,GO:0006281
....
....

（有些在逗号后面有空格，有些在逗号后面没有）。

我想只获取唯一标识符并删除第12列中每行的倍数：

GO:0042302
GO:0004386,GO:0005524,GO:0006281
....
....

这是我到目前为止所做的：

for row in `fileA`
do
    cut -f12 $row | sed "s/,/\n/" | sort | uniq | paste fileA - | \
    awk 'BEGIN {OFS=FS="\t"}{print $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $13}'
done > out

我的想法是一次遍历每一行，删除第12列，用换行符替换所有逗号，然后排序并使用uniq去除重复项，将其粘贴回来并以正确的顺序打印列，跳过原始标识符列。

但是，这似乎不起作用。有什么想法吗？

Answer 1

为了完整起见，并且因为我个人更喜欢Perl而不是Awk，这是一个Perl单线解决方案：

perl -F'\t' -le '%u=();@k=split/,/,$F[11];@u{@k}=@k;$F[11]=join",",sort
        keys%u;print join"\t",@F'

说明：

-F'\t'  Loop over input lines, splitting each one into fields at tabs
-l      automatically remove newlines from input and append on output
-e      get code to execute from the next argument instead of standard input

%u = ();                 # clear out the hash variable %u
@k = split /,/, $F[11];  # Split 12th field (1st is 0) on comma into array @k               
@u{@k} = @k;             # Copy the contents of @k into @u as key/value pairs

由于散列键是唯一的，因此最后一步意味着%u的键现在是@k的重复数据删除副本。

$F[11] = join ",", sort keys %u; # replace the 12th field with the sorted unique list
print join "\t", @F;             # and print out the modified line

Answer 2

如果我理解正确，那么请使用awk：

awk -F '\t' 'BEGIN { OFS = FS } { delete b; n = split($12, a, /, */); $12 = ""; for(i = 1; i <= n; ++i) { if(!(a[i] in b)) { b[a[i]]; $12 = $12 a[i] "," } } sub(/,$/, "", $12); print }' filename

其工作原理如下：

BEGIN { OFS = FS }           # output FS same as input FS
{
  delete b                   # clear dirty table from last pass

  n = split($12, a, /, */)   # split 12th field into tokens,
  $12 = ""                   # then clear it out for reassembly

  for(i = 1; i <= n; ++i) {  # wade through those tokens
    if(!(a[i] in b)) {       # those that haven't been seen yet:
      b[a[i]]                # remember that they were seen
      $12 = $12 a[i] ","     # append to result
    }
  }
  sub(/,$/, "", $12)         # remove trailing comma from resulting field

  print                      # print the transformed line
}

delete b;只有一段时间符合POSIX标准，所以如果你正在处理旧的awk并且它失败了，请参阅@ MarkReed的评论，以获得古代awk应该接受的另一种方式

Answer 3

使用字段2代替字段12：

$ cat tst.awk
BEGIN{ FS=OFS="\t" }
{
    split($2,f,/ *, */)
    $2 = ""
    delete seen
    for (i=1;i in f;i++) {
        if ( !seen[f[i]]++ ) {
            $2 = $2 (i>1?",":"") f[i]
        }
    }
    print
}

$ cat file
a,a,a   GO:0042302, GO:0042302, GO:0042302      b,b,b
c,c,c   GO:0004386,GO:0005524,GO:0006281, GO:0004386,GO:0005524,GO:0006281      d,d,d

$ awk -f tst.awk file
a,a,a   GO:0042302      b,b,b
c,c,c   GO:0004386,GO:0005524,GO:0006281        d,d,d

如果您的awk不支持delete seen，您可以使用split("",seen)。

Answer 4

使用此awk：

awk -F '\t' -v OFS='\t' '{
      delete seen;
      split($12, a, /[,; ]+/);
      for (i=1; i<=length(a); i++) {
         if (!(a[i] in seen)) {
            seen[a[i]];
            s=sprintf("%s%s,", s, a[i])
         }
      }
$12=s} 1' file
GO:0042302,
GO:0042302,GO:0004386,GO:0005524,GO:0006281,

Answer 5

在示例数据中，逗号后跟空格是第12个字段的分隔符。之后的每个子字段仅仅是第一个字段的重复。子字段似乎已按排序顺序排列。

GO:0042302, GO:0042302, GO:0042302
            ^^^dup1^^^  ^^^dup2^^^
GO:0004386,GO:0005524,GO:0006281, GO:0004386,GO:0005524,GO:0006281
                                  ^^^^^^^^^^^^^^^dup1^^^^^^^^^^^^^

基于此，您可以简单地保留第一个子字段并抛弃其余部分：

awk -F"\t" '{sub(/, .*/, "", $12)} 1' fileA

如果相反，您可以使用不同的重复子字段集，其中键的排序方式如下：

GO:0042302, GO:0042302, GO:0042302, GO:0062122,GO:0055000, GO:0055001, GO:0062122,GO:0055000
GO:0004386,GO:0005524,GO:0006281, GO:0005525, GO:0004386,GO:0005524,GO:0006281

如果您遇到默认的MacOS awk，可以在awk可执行脚本中引入sort / uniq函数：

#!/usr/bin/awk -f

BEGIN {FS="\t"}

{
    c = uniq(a, split($12, a, /, |,/))
    sort(a, c)
    s = a[1]
    for(i=2; i<=c; i++) { s = s "," a[i] }
    $2 = s
}

47  # print out the modified line

    # take an indexed arr as from split and de-dup it
function uniq(arr, len,      i, uarr) {
    for(i=len; i>=1; i--) { uarr[arr[i]] }
    delete arr
    for(k in uarr) { arr[++i] = k }
    return( i )
}

    # slightly modified from
    # http://rosettacode.org/wiki/Sorting_algorithms/Bubble_sort#AWK
function sort(arr, len,      haschanged, tmp, i)
{
    haschanged = 1
    while( haschanged==1 ) {
        haschanged = 0

        for(i=1; i<=(len-1); i++) {
            if( arr[i] > arr[i+1] ) {
                tmp = arr[i]
                arr[i] = arr[i + 1]
                arr[i + 1] = tmp
                haschanged = 1
            }
        }
    }
}

如果你有GNU-awk，我认为你可以用sort(a, c)换掉asort(a)来电，并完全放弃冒泡排序本地功能。

我在第12个字段中得到以下内容：

GO:0042302,GO:0055000,GO:0055001,GO:0062122
GO:0004386,GO:0005524,GO:0005525,GO:0006281

Unix / Bash：单元格上的Uniq

5 个答案: