我正在尝试过滤掉在grok的帮助下收到的日志。以下是样本日志
INFO | jvm 1 | main | 2013/04/05 01:08:47.048 | [m[32mINFO [TaskExecutor-master-2443-ProcessTask [31111111112]] [b2cConfirmationAction] CRON JOB ID : 101AA1C, ACTION : ConfirmationAction , CUSTOMER ID : 000001111111 , EMAIL ADDRESS : abc@gmail.com , SCHEDULE : Every 1 week , MESSAGE : Execution started for action ConfirmationAction
我在更新logstash conf文件之前使用grok调试器(https://grokdebug.herokuapp.com/)进行测试。 以下是我的过滤器代码:
%{LOGLEVEL:level}%{GREEDYDATA:greedydata}%{SPACE}%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}%{GREEDYDATA:gd} \[(?:%{WORD:action})\]%{GREEDYDATA:cronjobresult}
在这里,我正在以
为目标"level": [ [ "INFO" ] ], "greedydata": [ [ " | jvm 1 | main | 20" ] ], "SPACE": [ [ "", " " ] ], "YEAR": [ [ "13" ] ], "MONTHNUM": [ [ "04" ] ], "MONTHDAY": [ [ "05" ] ], "HOUR": [ [ "01" ] ], "MINUTE": [ [ "08" ] ], "SECOND": [ [ "47.048" ] ], "gd": [ [ " | \u001b[m\u001b[32mINFO [TaskExecutor-master-2443-ProcessTask [31111111112]]" ] ], "action": [ [ "b2cConfirmationAction" ] ], "cronjobresult": [ [ " CRON JOB ID : 101AA4A , ACTION : ConfirmationAction , CUSTOMER ID : 000001111111 , EMAIL ADDRESS : abc@gmail.com , SCHEDULE : Every 1 week , MESSAGE : Execution started for action ConfirmationAction" ] ] }
我的要求是获得cronjobresult下的值,如cron job iD客户ID,具有不同的独立字段,以便我可以在kibana中使用这些值。现在我无法得到它。此外,我已经两次使用greedyData,这个日志的更好的方法将是可观的。
答案 0 :(得分:1)
您可以进一步扩展过滤器并明确匹配。例如,要匹配cron作业ID,您可以在过滤器中编写CRON JOB ID : %{BASE16NUM:Cron_job_id}
。
如果您不需要日志中的任何数据,那么您只需编写.*
而不是GREEDYDATA
,它就会被跳过。
以下是日志的完整过滤器,
%{LOGLEVEL:level}%{GREEDYDATA:greedydata}%{SPACE}%{YEAR}[/-]%{MONTHNUM}[/-]%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}%{GREEDYDATA:gd} \[(?:%{WORD:action})\] CRON JOB ID : %{BASE16NUM:Cron_job_id},.*CUSTOMER ID : %{NUMBER:Customer_id}.*EMAIL ADDRESS : %{EMAILADDRESS}.*SCHEDULE : %{GREEDYDATA:schedule}.*, MESSAGE : %{GREEDYDATA:Message}
<强>输出:强>
{
"level": [
[
"INFO"
]
],
"greedydata": [
[
" | jvm 1 | main | 20"
]
],
"SPACE": [
[
"",
" "
]
],
"YEAR": [
[
"13"
]
],
"MONTHNUM": [
[
"04"
]
],
"MONTHDAY": [
[
"05"
]
],
"HOUR": [
[
"01"
]
],
"MINUTE": [
[
"08"
]
],
"SECOND": [
[
"47.048"
]
],
"gd": [
[
" | [m[32mINFO [TaskExecutor-master-2443-ProcessTask [31111111112]]"
]
],
"action": [
[
"b2cConfirmationAction"
]
],
"Cron_job_id": [
[
"101AA1C"
]
],
"Customer_id": [
[
"000001111111"
]
],
"BASE10NUM": [
[
"000001111111"
]
],
"EMAILADDRESS": [
[
"abc@gmail.com"
]
],
"local": [
[
"abc"
]
],
"remote": [
[
"gmail.com"
]
],
"schedule": [
[
"Every 1 week "
]
],
"Message": [
[
"Execution started for action"
]
]
}
请注意,我使用了https://github.com/rgevaert/grok-patterns/blob/master/grok.d/postfix_patterns
中的EMAILADDRESS
模式
如果您想在https://grokdebug.herokuapp.com上进行测试,则需要添加
EMAILADDRESSPART [a-zA-Z0-9_.+-=:]+
EMAILADDRESS %{EMAILADDRESSPART:local}@%{EMAILADDRESSPART:remote}
通过检查add custom patterns