如何收集IP和用户代理信息并使用AWK从nginx访问日志中获取它们的基本IP地址？

时间：2016-05-12 08:52:26

标签： linux bash shell nginx awk

我有一个示例日志文件

27.151.49.215 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1977 "http://localhost/" "Mozilla/5.0 (iPhone 6p; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/6.0 MQQBrowser/6.6.1 Mobile/12B411 Safari/8536.25"
49.73.31.190 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1813 "http://localhost/" "Mozilla/5.0 (iPhone 5SGLOBAL; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/6.0 MQQBrowser/6.6 Mobile/13B143 Safari/8536.25"
114.80.188.61 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 165 "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36"
223.64.63.228 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 2068 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13E238 QQ/6.2.0.427 Pixel/1080 NetType/WIFI Mem/48"
101.251.3.75 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1975 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/12F70 UCBrowser/10.9.14.779 Mobile"
101.251.3.75 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1975 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/12F70 UCBrowser/10.9.14.779 Mobile"
101.251.3.75 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1975 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/12F70 UCBrowser/10.9.14.779 Mobile"
101.251.3.75 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1975 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/12F70 UCBrowser/10.9.14.779 Mobile"
221.204.176.30 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 165 "http://localhost/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36"
222.77.208.177 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 2621 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Mobile/11D169 rabbit%2F1.0 baiduboxapp/0_0.0.8.6_enohpi_069_046/1.7_1C2%253enohPi/1099a/82840F498905C55D0EB7EBB0CF5DDC44BAF811E8FFCCOABNILE/1"
221.3.134.130 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 1962 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13E238 Safari/601.1"
123.157.71.167 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 2069 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/13E238 UCBrowser/10.9.13.779 Mobile"
39.187.201.169 - - [10/May/2016:23:59:59 +0800] "GET /m/index.php HTTP/1.1" 200 2621 "http://localhost/" "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2_1 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/13D15 UCBrowser/10.9.14.779 Mobile"

我想将所有ip和用户代理信息收集到一个文件中，并且unq相同的IP地址，我该如何处理awk？

输出如：

4 101.251.3.75 "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/12F70 UCBrowser/10.9.14.779 Mobile"
1 27.151.49.215 "Mozilla/5.0 (iPhone 6p; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/6.0 MQQBrowser/6.6.1 Mobile/12B411 Safari/8536.25"
1 49.73.31.190 "Mozilla/5.0 (iPhone 5SGLOBAL; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/6.0 MQQBrowser/6.6 Mobile/13B143 Safari/8536.25"
1 114.80.188.61 "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36"
1 223.64.63.228 "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Mobile/13E238 QQ/6.2.0.427 Pixel/1080 NetType/WIFI Mem/48"
1 221.204.176.30 "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36"
1 22.77.208.177 "Mozilla/5.0 (iPhone; CPU iPhone OS 7_1 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Mobile/11D169 rabbit%2F1.0 baiduboxapp/0_0.0.8.6_enohpi_069_046/1.7_1C2%253enohPi/1099a/82840F498905C55D0EB7EBB0CF5DDC44BAF811E8FFCCOABNILE/1"
1 221.3.134.130 "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13E238 Safari/601.1"
1 123.157.71.167 "Mozilla/5.0 (iPhone; CPU iPhone OS 9_3_1 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/13E238 UCBrowser/10.9.13.779 Mobile"
1 39.187.201.169 "Mozilla/5.0 (iPhone; CPU iPhone OS 9_2_1 like Mac OS X; zh-CN) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/13D15 UCBrowser/10.9.14.779 Mobile"

2 个答案:

答案 0 :(得分：0)

awk -F"[\"]" '{
    sub(/-.*$/,"",$1);
    a[$1" \""$(NF-1)"\""]++
}
END {
    for ( i in a) print a[i],i
}' logfile

$1将拥有IP，$(NF-1)将拥有代理商信息。 使用这些值作为索引创建数组a，并为每次出现类似索引值时增加此数组的值。

答案 1 :(得分：0)

使用sed，sort和uniq：

sed 's/\([^ ]*\).* \("[^"]*"\)/\1 \2/' file | sort | uniq -c
相关问题
最新问题