使用grep在日志文件中获取唯一条目

时间：2016-05-02 12:46:57

标签： bash logging grep

我有以下日志文​​件

2016-05-02 11:37:32,211 (glastopf.glastopf) 190.244.96.69 requested GET /phpMyAdmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:33,065 (glastopf.glastopf) 190.244.96.69 requested GET /pma/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:33,990 (glastopf.glastopf) 190.244.96.69 requested GET /myadmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:55:07,672 (glastopf.glastopf) 141.8.83.213 requested GET / on 04680bb24791:80
2016-05-02 11:55:07,719 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:07,843 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:09,195 (glastopf.glastopf) 141.8.83.213 requested POST /index on 04680bb24791:80
2016-05-02 11:55:09,233 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:09,269 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:10,899 (glastopf.glastopf) 141.8.83.213 requested POST /index on 04680bb24791:80
2016-05-02 11:55:10,935 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:10,970 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:13,307 (glastopf.glastopf) 141.8.83.213 requested POST /comments on 04680bb24791:80
2016-05-02 11:55:13,359 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:13,378 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:28,106 (glastopf.glastopf) 141.8.27.157 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,193 (glastopf.glastopf) 31.13.102.123 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,205 (glastopf.glastopf) 141.8.27.157 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:28,381 (glastopf.glastopf) 173.252.120.102 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,512 (glastopf.glastopf) 141.8.27.157 requested GET /favicon.ico on 04680bb24791:80

我想得到唯一的条目，丢弃前三个属性，即日期时间和glastopf glastopf。我希望从IP开始获得独特的entrie。我有以下grep命令：

sudo grep 'GET\|POST' /home/tsec/prototype/logs/glastopf.log | \
     sort -k4,4 | tac | sort -k4,4 | sort -k1,2 | \
     tail -n 20 > /home/tsec/prototype/logs/extractedlogs/glastopfresult.log

排序中一定有问题

1 个答案:

答案 0 :(得分：4)

这是sort的任务：

sort -uk4 file.log

如果您希望最后一个条目使用tac来反转这些行并执行操作，那么当从第4个字段开始多行相同时，这将获得第一行：

tac file.log | sort -uk4

示例： 

$ cat file.txt
2016-05-02 11:37:32,211 (glastopf.glastopf) 190.244.96.69 requested GET /phpMyAdmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:33,065 (glastopf.glastopf) 190.244.96.69 requested GET /pma/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:33,990 (glastopf.glastopf) 190.244.96.69 requested GET /myadmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:55:07,672 (glastopf.glastopf) 141.8.83.213 requested GET / on 04680bb24791:80
2016-05-02 11:55:07,719 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:07,843 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:09,195 (glastopf.glastopf) 141.8.83.213 requested POST /index on 04680bb24791:80
2016-05-02 11:55:09,233 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:09,269 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:10,899 (glastopf.glastopf) 141.8.83.213 requested POST /index on 04680bb24791:80
2016-05-02 11:55:10,935 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:10,970 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:13,307 (glastopf.glastopf) 141.8.83.213 requested POST /comments on 04680bb24791:80
2016-05-02 11:55:13,359 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:13,378 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:28,106 (glastopf.glastopf) 141.8.27.157 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,193 (glastopf.glastopf) 31.13.102.123 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,205 (glastopf.glastopf) 141.8.27.157 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:28,381 (glastopf.glastopf) 173.252.120.102 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,512 (glastopf.glastopf) 141.8.27.157 requested GET /favicon.ico on 04680bb24791:80

$ sort -uk4 file.txt
2016-05-02 11:55:28,512 (glastopf.glastopf) 141.8.27.157 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:28,106 (glastopf.glastopf) 141.8.27.157 requested GET / on 04680bb24791:80
2016-05-02 11:55:28,205 (glastopf.glastopf) 141.8.27.157 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:07,843 (glastopf.glastopf) 141.8.83.213 requested GET /favicon.ico on 04680bb24791:80
2016-05-02 11:55:07,672 (glastopf.glastopf) 141.8.83.213 requested GET / on 04680bb24791:80
2016-05-02 11:55:07,719 (glastopf.glastopf) 141.8.83.213 requested GET /style.css on 04680bb24791:80
2016-05-02 11:55:13,307 (glastopf.glastopf) 141.8.83.213 requested POST /comments on 04680bb24791:80
2016-05-02 11:55:09,195 (glastopf.glastopf) 141.8.83.213 requested POST /index on 04680bb24791:80
2016-05-02 11:55:28,381 (glastopf.glastopf) 173.252.120.102 requested GET / on 04680bb24791:80
2016-05-02 11:37:33,990 (glastopf.glastopf) 190.244.96.69 requested GET /myadmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:32,211 (glastopf.glastopf) 190.244.96.69 requested GET /phpMyAdmin/scripts/setup.php on 04680bb24791:80
2016-05-02 11:37:33,065 (glastopf.glastopf) 190.244.96.69 requested GET /pma/scripts/setup.php on 04680bb24791:80
2016-05-02 11:55:28,193 (glastopf.glastopf) 31.13.102.123 requested GET / on 04680bb24791:80
相关问题
最新问题