Question

我有一个java应用程序日志文件，其中包含我想在不同行上选择的一些参数，我希望所有这些都在一行上。

日志示例：

<td class="bold">25</td>

过滤示例1：

2017-05-31 16:30:38.889 [http-listener-1(2)] INFO  com.domain.api.provider.ClientRequestDataFilter - ContainerRequestContextFilter start
2017-05-31 16:30:38.889 [http-listener-1(2)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client IP Address : 172.25.4.32
2017-05-31 16:30:38.889 [http-listener-1(2)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client User-Agent : Dalvik/2.1.0 (Linux; U; Android 7.1.2; Nexus 5X Build/N2G47O)
2017-05-31 16:30:38.889 [http-listener-1(2)] INFO  com.domain.api.provider.ClientRequestDataFilter - ContainerRequestContextFilter end
2017-05-31 16:30:38.891 [http-listener-1(2)] INFO  com.domain.api.controller.MyDataController - Start getMyPageResponse method, input = {"token":"AppleBallCatDogElephantFoxGoatHorse"}
2017-05-31 16:30:38.891 [http-listener-1(2)] INFO  com.domain.biz.service.impl.MyTestService - Start getMyPageInfo method, requestDto = TokenRequestDto [token=AppleBallCatDogElephantFoxGoatHorse]
2017-05-31 16:30:38.891 [http-listener-1(2)] INFO  com.domain.biz.logic.impl.UserAuthBizLogic - Start authentication token=AppleBallCatDogElephantFoxGoatHorse
2017-05-31 16:30:38.895 [http-listener-1(2)] INFO  com.domain.biz.logic.impl.UserAuthBizLogic - End authentication, UserInfo=UserInfoDto[userID=123456789]
2017-05-31 16:30:38.895 [http-listener-1(2)] DEBUG com.domain.biz.service.impl.MyTestService - userID = 123456789, token = AppleBallCatDogElephantFoxGoatHorse
2017-05-31 16:30:39.422 [http-listener-1(2)] INFO  com.domain.biz.service.impl.MyTestService - End getMyPageInfo method
2017-05-31 16:30:39.423 [http-listener-1(3)] INFO  com.domain.biz.service.impl.MyTestService - Start getMyPageResponse method
2017-05-31 16:30:39.423 [http-listener-1(2)] INFO  com.domain.biz.service.impl.MyTestService - End getMyPageResponse method
2017-05-31 16:30:39.423 [http-listener-1(2)] INFO  com.domain.api.controller.MyDataController - End getMyPageResponse method, Total Computation/Execution Time : 0.532s

2017-05-31 16:42:36.500 [http-listener-1(3)] INFO  com.domain.api.provider.ClientRequestDataFilter - ContainerRequestContextFilter start
2017-05-31 16:42:36.501 [http-listener-1(3)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client IP Address : 172.25.4.32
2017-05-31 16:42:36.501 [http-listener-1(3)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client User-Agent : MyiOSApp/v2.2.2 (build:888; iOS 10.3.2) Alamofire/4.4.0
2017-05-31 16:42:36.501 [http-listener-1(3)] INFO  com.domain.api.provider.ClientRequestDataFilter - ContainerRequestContextFilter end
2017-05-31 16:42:36.506 [http-listener-1(3)] INFO  com.domain.api.controller.MyDataController - Start getMyPageResponse method, input = {"token":"KingLionMonkeyNestOwlParrotQueenRabbitSnakeTiger"}
2017-05-31 16:42:36.507 [http-listener-1(3)] INFO  com.domain.biz.logic.impl.UserAuthBizLogic - Start authentication token=KingLionMonkeyNestOwlParrotQueenRabbitSnakeTiger
2017-05-31 16:42:36.507 [http-listener-1(3)] INFO  com.domain.biz.service.impl.MyTestService - Start getMyPageInfo method
2017-05-31 16:42:36.515 [http-listener-1(3)] INFO  com.domain.biz.logic.impl.UserAuthBizLogic - End authentication, UserInfo=UserInfoDto[userID=987654321]
2017-05-31 16:42:36.515 [http-listener-1(3)] DEBUG com.domain.biz.service.impl.MyTestService - userID = 987654321, token = KingLionMonkeyNestOwlParrotQueenRabbitSnakeTiger
2017-05-31 16:42:36.743 [http-listener-1(3)] INFO  com.domain.biz.service.impl.MyTestService - End getMyPageInfo method
2017-05-31 16:42:36.743 [http-listener-1(3)] INFO  com.domain.biz.service.impl.MyTestService - Start getMyPageResponse method
2017-05-31 16:42:36.743 [http-listener-1(3)] INFO  com.domain.biz.service.impl.MyTestService - End getMyPageResponse method
2017-05-31 16:42:36.743 [http-listener-1(3)] INFO  com.domain.api.controller.MyDataController - End getMyPageResponse method, Total Computation/Execution Time : 0.237s

过滤示例2：

cat /usr/local/sun/glassfish/domains/universal-api/logs/my-new-api/system.log | grep -e "Filter" -e "userID"

2017-05-31 16:30:38.889 [http-listener-1(2)] INFO  com.domain.api.provider.ClientRequestDataFilter - ContainerRequestContextFilter start                                                                                                     
2017-05-31 16:30:38.889 [http-listener-1(2)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client IP Address : 172.25.4.32                                                                                                         
2017-05-31 16:30:38.889 [http-listener-1(2)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client User-Agent : MyiOSApp/v2.2.2 (build:888; iOS 10.3.2) Alamofire/4.4.0                                                       
2017-05-31 16:30:38.889 [http-listener-1(2)] INFO  com.domain.api.provider.ClientRequestDataFilter - ContainerRequestContextFilter end                                                                                                       
2017-05-31 16:30:38.895 [http-listener-1(2)] INFO  com.domain.biz.logic.impl.UserAuthBizLogic - End authentication, UserInfo=UserInfoDto[userID=123456789]                                                                                   
2017-05-31 16:30:38.895 [http-listener-1(2)] DEBUG com.domain.biz.service.impl.MyTestService - userID = 123456789, token = AppleBallCatDogElephantFoxGoatHorse                                                                               
2017-05-31 16:42:36.500 [http-listener-1(3)] INFO  com.domain.api.provider.ClientRequestDataFilter - ContainerRequestContextFilter start                                                                                                     
2017-05-31 16:42:36.501 [http-listener-1(3)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client IP Address : 172.25.4.32                                                                                                         
2017-05-31 16:42:36.501 [http-listener-1(3)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client User-Agent : Dalvik/2.1.0 (Linux; U; Android 7.1.2; Nexus 5X Build/N2G47O)                                                       
2017-05-31 16:42:36.501 [http-listener-1(3)] INFO  com.domain.api.provider.ClientRequestDataFilter - ContainerRequestContextFilter end                                                                                                       
2017-05-31 16:42:36.515 [http-listener-1(3)] INFO  com.domain.biz.logic.impl.UserAuthBizLogic - End authentication, UserInfo=UserInfoDto[userID=987654321]                                                                                   
2017-05-31 16:42:36.515 [http-listener-1(3)] DEBUG com.domain.biz.service.impl.MyTestService - userID = 987654321, token = KingLionMonkeyNestOwlParrotQueenRabbitSnakeTiger

我想从这些过滤掉4件事：

时间，IP地址，用户代理，用户ID，令牌

预期输出样本：

cat /usr/local/sun/glassfish/domains/universal-api/logs/my-new-api/system.log | grep -e "IP Address" -e "User-Agent" -e "userID"

2017-05-31 16:30:38.889 [http-listener-1(2)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client IP Address : 172.25.4.32                                                                                                         
2017-05-31 16:30:38.889 [http-listener-1(2)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client User-Agent : MyiOSApp/v2.2.2 (build:888; iOS 10.3.2) Alamofire/4.4.0                                                       
2017-05-31 16:30:38.895 [http-listener-1(2)] INFO  com.domain.biz.logic.impl.UserAuthBizLogic - End authentication, UserInfo=UserInfoDto[userID=123456789]                                                                                   
2017-05-31 16:30:38.895 [http-listener-1(2)] DEBUG com.domain.biz.service.impl.MyTestService - userID = 123456789, token = AppleBallCatDogElephantFoxGoatHorse                                                                               
2017-05-31 16:42:36.501 [http-listener-1(3)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client IP Address : 172.25.4.32                                                                                                         
2017-05-31 16:42:36.501 [http-listener-1(3)] INFO  com.domain.api.provider.ClientRequestDataFilter - Client User-Agent : Dalvik/2.1.0 (Linux; U; Android 7.1.2; Nexus 5X Build/N2G47O)                                                       
2017-05-31 16:42:36.515 [http-listener-1(3)] INFO  com.domain.biz.logic.impl.UserAuthBizLogic - End authentication, UserInfo=UserInfoDto[userID=987654321]                                                                                   
2017-05-31 16:42:36.515 [http-listener-1(3)] DEBUG com.domain.biz.service.impl.MyTestService - userID = 987654321, token = KingLionMonkeyNestOwlParrotQueenRabbitSnakeTiger

并将它们全部打印在一行中。

它们将始终以相同的顺序出现。

为了协助选择文字，我添加了2017-05-31 16:30:38.889, 172.25.4.32, MyiOSApp/v2.2.2 (build:888; iOS 10.3.2) Alamofire/4.4.0, 123456789 2017-05-31 16:30:38.889, 172.25.4.32, Dalvik/2.1.0 (Linux; U; Android 7.1.2; Nexus 5X Build/N2G47O), 987654321和ContainerRequestContextFilter start

我相信 ContainerRequestContextFilter end ， grep ， awk ＆amp; sed 可以帮助我完成这项工作，还有其他有效方法吗？

更新：问题是多个用户一个接一个地有很多此类条目。我想知道编写shell脚本，python代码或类似技术会更有效。

我可以在多行日志上使用awk只获得1行，我需要每个用户1行。

Answer 1

gawk 解决方案：

awk '!f && /ContainerRequestContextFilter/{ f=1; t=$1 FS $2; next }
     /Client IP Address/{ ip=$NF; next }
     /Client User-Agent/{ uagent=$10; for(i=11;i<=NF;i++) uagent=uagent FS $i; next }
     /userID =.* token =/{ match($0,/userID = ([0-9]+), token = (.+)/,a); f=0; 
      printf("%s, %s, %s, %s, %s\n",t,ip,uagent,a[1],a[2]); next}
' system.log

输出：

2017-05-31 16:30:38.889, 172.25.4.32, Dalvik/2.1.0 (Linux; U; Android 7.1.2; Nexus 5X Build/N2G47O), 123456789, AppleBallCatDogElephantFoxGoatHorse
2017-05-31 16:42:36.500, 172.25.4.32, MyiOSApp/v2.2.2 (build:888; iOS 10.3.2) Alamofire/4.4.0, 987654321, KingLionMonkeyNestOwlParrotQueenRabbitSnakeTiger

!f && /ContainerRequestContextFilter/ - 处理起始ContainerRequestContextFilter行
/Client IP Address/{ ip=$NF; next } - 包含Client IP Address值的处理行
/Client User-Agent/ - 处理行匹配Client User-Agent
/userID =.* token =/ - 处理行匹配userID =和token =

Answer 2

我想您已经将cat /usr/local/sun/glassfish/domains/universal-api/logs/my-new-api/system.log | grep -e "Filter" -e "userID"的输出定向到名为log的文件。

然后您可以尝试按照awk命令来实现目标，

awk '
  /ContainerRequestContextFilter start/ {printf "%s %s,",$1,$2}
  /Client User-Agent/ {for(i=10;i<=NF;i++) printf "%s ",$i}
  /userID/ {printf ", %s",$9}
  /token/ {printf " %s\n",$12}
' log

输出应符合您的要求。

将来自多行日志的信息组合到单行

2 个答案: