Question

我正在使用ELK（filebeat，logstash，elasticsearch，kibana）来进行日志管理。在一个日志文件中，我有三种格式。我正在使用

在一种格式中，我有日期+参数+ json + stacktrace。这种格式文本有多行。在第二种格式中，它只是日期+ requestMethod（获取或发布）+一些文本。它在一条线上。在第三种格式中，它有日期+模块名称:(在这种情况下，它是paymentAdmin）+ json

我想我可以使用logStash通过if，else if子句和3种匹配器来处理这三种格式。有人可以告诉我如何编写if，else if子句在一个日志文件中处理3种格式吗？

2018-04-10 10:14:18 params:{"currentPage":"1","schoolname":"","schoolcode":"","pageSize":"10","pageNumber":"1","ispage":"yes","sortOrder":"asc","_":"1523326435639"} java.lang.NullPointerException: null
    at com.datalook.group.BusinessHandler.queryGroup(BusinessHandler.java:204) ~[classes/:?]
    at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:43) [classes/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

121.251.19.243 - - [02/Jan/2018:00:00:13 +0800] "GET /eams5-student/static/cryptoJS/sha1.js HTTP/1.1" 404 - "http://jxglstu.hfut.edu.cn/eams5-student/login" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; rv:11.0) like Gecko" 0.0.0.107 80 0 - "101.226.226.221" "-"


2018-03-28 13:23:01 PaymentAdmin:{"manageamt":"0.00","managefee":0.2,"pcode":"01","txamt":"0.01"}

Answer 1

有多种方法可以实现，您可以定义单个grok过滤器，并匹配多个模式，类似这样，

filter {
   grok {
     match => [ "message", "PATTERN1", "PATTERN2" ]
    }
}

，或者

if "SUCCESS" not in [tags]{
  grok {
    match => { "message" => PATTERN1}
    add_tag => ["SUCCESS"]
    remove_tag => ["_grokparsefailure"]
  }
}

if "SUCCESS" not in [tags]{
  grok {
    match => { "message" => PATTERN2}
    add_tag => ["SUCCESS"]
    remove_tag => ["_grokparsefailure"]
  }
}

或者，setting a type through input plugin

的条件方法

if [type] == "syslog" {
  grok {
    match => ["message", "PATTERN1"]
  }
}

else if [type] == "nginx" {
      grok {  
        match => ["message", "PATTERN2"]
      }
  }

上述方法的问题在于，模式将按顺序应用，直到找到匹配为止。

Logstash在一个日志文件中处理多种格式的文本

1 个答案: