Question

我有一个示例日志文件，如sample.txt所示：这些错误是相同的，但时间戳和错误代码不同

catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:2017-08-19 15:43:59.063 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514880 Listener={set}
catalina.out:2017-08-19 15:43:59.703 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514896 Listener={set}
catalina.out:2017-08-19 16:19:41.274 [1526172869 tomcat-http--1] [TEST ECHO]: [J] Connection SetExceptionListener conn=4515735 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)

有什么办法我可以grep或对此进行过滤并获得带有uniq错误行示例预期的输出

的输出

catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)

尝试

grep  Example.txt | uniq > out.txt

但是没用

日志文件的内容可能会有所不同，但想法是grep用于uniq错误行，而忽略了重复行

Answer 1

这是一个开始：

$ awk '{err=$0; gsub(/[0-9]+/,0,err)} !seen[err]++' file
catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:2017-08-19 16:19:41.274 [1526172869 tomcat-http--1] [TEST ECHO]: [J] Connection SetExceptionListener conn=4515735 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)

向err添加更多更改，以按照您认为合适的方式对错误消息的各个部分进行规范化，例如：

$ awk '{err=$0; gsub(/[0-9]+/,0,err); gsub(/\[[^]]+]/,"[]",err)} !seen[err]++' file
catalina.out:2017-08-19 15:43:58.991 [2012293659 localhost] [TEST ECHO]: [J] Connection SetExceptionListener conn=4514879 Listener={set}
catalina.out:javax.jms.TransactionRolledBackException: Commit failed due to prior failure or after fault-tolerant switch, transaction rolled back
catalina.out: at com.tibco.tibjms.Testjmsx.buildException(Testjmsx.java:601)

Answer 2

grep -o "pattern" yourfile |sort |uniq

排序是必要的，除非您感兴趣的重复行紧挨着出现

从日志文件中过滤出uniq行

2 个答案: