Question

我将Web和API日志结合在一起，我想将其分别保存在elasticsearch中。因此，我想编写一种模式，如果请求是针对API的，那么如果过去应该执行，请求是Web，则应该执行日志的一部分。

下面是一些Web和API日志。

00:06:27,778 INFO  [stdout] (ajp--0.0.0.0-8009-38) 00:06:27.777 [ajp--0.0.0.0-8009-38] INFO  c.r.s.web.rest.WidgetController - Method getWidgetDetails() started to get widget details.
00:06:27,783 INFO  [stdout] (ajp--0.0.0.0-8009-38) ---> HTTP GET http://api.survey.me/v1/getwidgetdetails?profileName=jeremy-steffens&profileLevel=INDIVIDUAL&companyProfileName=premier-nationwide-lending&hideHistory=true
00:06:27,817 INFO  [stdout] (ajp--0.0.0.0-8009-38) <--- HTTP 200 http://api.survey.me/v1/getwidgetdetails?profileName=jeremy-steffens&profileLevel=INDIVIDUAL&companyProfileName=premier-nationwide-lending&hideHistory=true (29ms)
00:06:27,822 INFO  [stdout] (ajp--0.0.0.0-8009-38) 00:06:27.822 [ajp--0.0.0.0-8009-38] INFO  c.r.s.web.rest.WidgetController - Method getWidgetDetails() finished.
00:06:27,899 INFO  [stdout] (ajp--0.0.0.0-8009-40) 00:06:27.899 [ajp--0.0.0.0-8009-40] INFO  c.r.s.web.controller.LoginController - Inside initLoginPage() of LoginController

我试图写条件，但是没有用。它仅根据线程名称起作用。在线程之后，我有多个类型的日志，因此如果有条件就无法写出。

(?:%{TIME:CREATED_ON})(?:%{SPACE})%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)

有人可以给我建议吗？

Answer 1

您无需使用if/else条件，您可以使用多种模式，一种将与API日志行匹配，而另一种将与WEB日志行匹配。

对于API日志行，您可以使用以下模式：

(?:%{TIME:CREATED_ON})(?:%{SPACE})%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)%{SPACE}(?:%{DATA})%{SPACE}\[%{DATA}\]%{SPACE}%{WORD}%{SPACE}%{GREEDYDATA:MSG}

您的回报将是这样的：

{
"MSG": "c.r.s.web.controller.LoginController - Inside initLoginPage() of LoginController",
"CREATED_ON": "00:06:27,899",
"LEVEL": "INFO",
"THREAD": "ajp--0.0.0.0-8009-40"
}

对于网络行，您可以使用以下模式：

(?:%{TIME:CREATED_ON})(?:%{SPACE})%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)%{SPACE}%{DATA}%{WORD:PROTOCOL}%{SPACE}%{WORD:MethodOrStatus}%{SPACE}%{GREEDYDATA:ENDPOINT}

结果将是：

{
"CREATED_ON": "00:06:27,783",
"PROTOCOL": "HTTP",
"ENDPOINT": "http://api.survey.me/v1/getwidgetdetails?profileName=jeremy-steffens&profileLevel=INDIVIDUAL&companyProfileName=premier-nationwide-lending&hideHistory=true",
"LEVEL": "INFO",
"THREAD": "ajp--0.0.0.0-8009-38",
"MethodOrStatus": "GET"
}

要在grok中使用多种模式，只需执行以下操作：

grok {
  match => ["message", "pattern1", "pattern2"]
}

或者您可以将模式保存到文件中，并使用patterns_dir指向文件目录。

如果您仍然想使用条件语句，只需检查消息中是否有任何内容，例如：

if "HTTP" in [message] { 
 grok { your grok for the web messages }
} else {
 grok { your grok for the api messages }
}

Answer 2

我没有收到任何错误，但是弹性搜索中仍然没有创建索引。下面是我的配置文件。

input {
  beats {
    port => 5044
  }
}
filter {
    if [log_type] == "apache-apis" {
        grok {
            match => { "message" => "^%{IP:CLIENT_IP} (?:-|%{USER:IDEN}) (?:-|%{USER:AUTH}) \[%{HTTPDATE:CREATED_ON}\] \"(?:%{WORD:REQUEST_METHOD} (?:/|%{NOTSPACE:REQUEST})(?: HTTP/%{NUMBER:HTTP_VERSION})?|-)\" %{NUMBER:RESPONSE_CODE} (?:-|%{WORD:BYTES}) (?:-|%{WORD:EXECUTION_TIME})"}
            add_field => {
                "LOG_TYPES" => "api-log"
            }
        overwrite => [ "message" ]
        }
    }
    if [log_type] == "apache-webs" {
        grok {
            match => { "message" => "%{HTTPDATE:CREATED_ON}%{NOTSPACE}%{SPACE} (?:-|%{IP:CLIENT_IP})%{SPACE} %{NOTSPACE}(?:-|%{WORD:REQUEST_METHOD}%{SPACE}) (?:-|%{NOTSPACE:REQUEST})(?: HTTP/%{NUMBER:HTTP_VERSION})%{NOTSPACE}(?:-|%{GREEDYDATA:OTHER_INFO}) (?:-|%{NUMBER:RESPONSE_CODE}) (?:-|%{WORD:BYTES}) (?:-|%{WORD:EXECUTION_TIME})"}
            add_field => {
                "LOG_TYPES" => "web-log"
            }
        overwrite => [ "message" ]
        }
    }
    if [log_type] == "jboss-apis" {
        grok {
            match => { "message" => "%{TIME:CREATED_ON}%{SPACE}%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE:URI_CLASS}\]%{SPACE}\(%{NOTSPACE:THREAD}\)(?<MESSAGE_LOG>[^\r\n]+)((\r?\n)(?<MORE-INFO>(.|\r?\n)+))?"}
            add_field => {
                "LOG_TYPES" => "jboss-api"
            }
        overwrite => [ "message" ]
        }
    }
    if [log_type] == "jboss-webs" {
        grok {
            match => { "message" => "(?:%{TIME:CREATED_ON})(?:%{SPACE})(?:%{WORD:LEVEL})%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)%{SPACE}(?:%{TIME})(?:%{SPACE})%{SPACE}\[%{NOTSPACE}\]%{SPACE}(?:%{SPACE})%{WORD:LEVEL}%{SPACE}%{JAVACLASS:CLASS} - (?<MESSAGE_LOG>[^\r\n]+)((\r?\n)(?<extra>(.|\r?\n)+))?"}
            add_field => {
                "LOG_TYPES" => "jboss-web"
            }
        }
        grok {
            match => { "message" => "(?:%{TIME:CREATED_ON})(?:%{SPACE})(?:%{WORD:LEVEL})%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:PROTOTYPE}%{SPACE}(?:%{NOTSPACE:STATUS})(?:%{SPACE})(?:%{URI:URI_CLASS})"}
            add_field => {
                "LOG_TYPES" => "jboss-web"
            }
        }
        grok {
            match => { "message" => "%{TIME:CREATED_ON}%{SPACE}%{WORD:LEVEL}%{SPACE}\[%{NOTSPACE}\]%{SPACE}\(%{NOTSPACE:THREAD}\)\ (?<MESSAGE_LOG>[^\r\n]+)\n%{SPACE}%{NOTSPACE}%{SPACE}%{JAVACLASS:URI-CLASS}"}
            add_field => {
                "LOG_TYPES" => "jboss-web"
            }
        }
    }
}
output {
    if [log_type] == "apache-apis" or [log_type] == "apache-webs" {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "apache-server-logs"
        }
    }
    if [log_type] == "jboss-apis" or [log_type] == "jboss-webs" {
        elasticsearch {
        hosts => ["localhost:9200"]
        manage_template => false
        index => "jboss-server-logs"
        }
    }
  stdout { codec => rubydebug } 
}

filebeat.yml conf文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/user/path/originallogs/apache/api/a-api.log
  fields:
    log_type: apache-apis
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /home/user/path/originallogs/apache/web/a-web.log
  fields:
    log_type: apache-webs
  fields_under_root: true

- type: log
  enabled: true
  paths:
    - /home/user/path/originallogs/jboss/api/jboss-api-log.log
  fields:
    log_type: jboss-apis
  fields_under_root: true


- type: log
  enabled: true
  paths:
    - /home/user/path/originallogs/jboss/web/jboss-web-log.log
  fields:
    log_type: jboss-webs
  fields_under_root: true

  #exclude_lines: ['^DBG']
  #include_lines: ['^ERR', '^WARN']

如何在logstash的grok模式中使用IF ELSE条件

2 个答案: