Question

我遇到Logstash配置问题我的日志模式是

2017-07-26 14:31:03,644 INFO  [http-bio-10.60.2.21-10267-exec-92] jsch.DeployManagerFileUSImpl (DeployManagerFileUSImpl.java:132) - passage par ficher temporaire .bindings.20170726-143103.tmp

我目前的模式是

match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} \(%{DATA:class}\):%{GREEDYDATA:message}" }

[http-bio-10.60.2.21-10267-exec-92]和jsch.DeployManagerFileUSImpl的哪种模式？

Answer 1

您所显示的当前模式似乎不起作用，因为您的示例消息中没有匹配\(%{DATA:class}\):%{GREEDYDATA:message}的任何内容，并且您没有在日志级别之后处理双空格。

如果你想匹配一行中间的一些随机内容，请使用％{DATA}，例如：

\ [％{DATA：MyField的} \]

然后你可以使用％{GREEDYDATA}来获取行尾的内容：

\ [％{DATA：myfield1} \]％{GREEDYDATA：myfield2}

如果您需要将这些项目分解为自己的字段，那么请更具体地使用该模式或使用第二个grok {}块。

Answer 2

在我的logstash.conf中，我将模式更改为

match => [ "message", "%{TIMESTAMP_ISO8601:logdate},%{INT} %{LOGLEVEL:log-level}  \[(?<threadname>[^\]]+)\] %{JAVACLASS:package} \(%{JAVAFILE:file}:%{INT:line}\) - %{GREEDYDATA:message}" ]

借助网站https://grokdebug.herokuapp.com/的帮助。

但我在kibana 5.4.3中看不到我的静态日志文件包含在/ home / elasticsearch / static_logs /目录中？

我的logstash配置文件带有“静态”部分

input { 
    file {
        type => "access-log"
         path => "/home/elasticsearch/tomcat/logs/*.txt"
    }
    file {
        type => "tomcat"
        path => "/home/elasticsearch/tomcat/logs/*.log" exclude => "*.zip"
        codec => multiline {
          negate => true
          pattern => "(^%{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM))"
          what => "previous"
        }
    }
        file {
        type => "static"
        path => "/home/elasticsearch/static_logs/*.log" exclude => "*.zip"
    }
}

filter {
    if [type] == "access-log" {
        grok {
            # Access log pattern is %a %{waffle.servlet.NegotiateSecurityFilter.PRINCIPAL}s %t %m %U%q %s %B %T &quot;%{Referer}i&quot; &quot;%{User-Agent}i&quot;
            match => [ "message" , "%{IPV4:clientIP} %{NOTSPACE:user} \[%{DATA:timestamp}\] %{WORD:method} %{NOTSPACE:request} %{NUMBER:status} %{NUMBER:bytesSent} %{NUMBER:duration} \"%{NOTSPACE:referer}\" \"%{DATA:userAgent}\"" ]
            remove_field => [ "message" ]
        }
        grok{
            match => [ "request", "/%{USERNAME:app}/" ]
            tag_on_failure => [ ]
        }
        date {
            match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
            remove_field => [ "timestamp" ]
        }
        geoip {
            source => ["clientIP"]
        }
        dns {
            reverse => [ "clientIP" ]
        }
        mutate {
            lowercase => [ "user" ]
            convert => [ "bytesSent", "integer", "duration", "float" ]
        }
        if [referer] == "-" {
            mutate {
                remove_field => [ "referer" ]
            }
        }
        if [user] == "-" {
            mutate {
                remove_field => [ "user" ]
            }
        }
    }
    if [type] == "tomcat" {
        if [message] !~ /(.+)/  {
            drop { }
        }
        grok{
             patterns_dir => "./patterns"
             overwrite => [ "message" ]
            # oK Catalina normal
             match => [ "message", "%{CATALINA_DATESTAMP:timestamp} %{NOTSPACE:className} %{WORD:methodName}\r\n%{LOGLEVEL: logLevel}: %{GREEDYDATA:message}" ]
        }
        grok{
            match => [ "path", "/%{USERNAME:app}.20%{NOTSPACE}.log"]
            tag_on_failure => [ ]
        }
        # Aug 25, 2014 11:23:31 AM
        date{
            match => [ "timestamp", "MMM dd, YYYY hh:mm:ss a" ]
            remove_field => [ "timestamp" ]
        }
    }
        if [type] == "static" {
        if [message] !~ /(.+)/  {
            drop { }
        }
        grok{
             patterns_dir => "./patterns"
             overwrite => [ "message" ]
            # 2017-08-03 16:01:11,352 WARN  [Thread-552] pcf2.AbstractObjetMQDAO (AbstractObjetMQDAO.java:137) - Descripteur de
            match => [ "message", "%{TIMESTAMP_ISO8601:logdate},%{INT} %{LOGLEVEL:log-level}  \[(?<threadname>[^\]]+)\] %{JAVACLASS:package} \(%{JAVAFILE:file}:%{INT:line}\) - %{GREEDYDATA:message}" ]
        }
        # 2017-08-03 16:01:11,352
        date{
            match => [ "timestamp", "YYYY-MM-dd hh:mm:ss,SSS" ]
            remove_field => [ "timestamp" ]
        }
    }
}

output {
    elasticsearch { hosts => ["192.168.99.100:9200"]}

}

我的错误在哪里？此致

logstash配置文件tomcat出错

2 个答案:

我的logstash配置文件带有“静态”部分