Question

2016-06-02 17:00:32<ERROR><CommonsHTTPTransportSender:361>: 
org.apache.abcd2.abcdFault
    at org.apache.abcd2.abcdFault.makeFault(abcdFault.java:430)
    at org.apache.abcd2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:83)
    at org.apache.abcd2.transport.http.CommonsHTTPTransportSender.sendUsingOutputStream(CommonsHTTPTransportSender.java:358)
    at java.lang.Thread.run(Thread.java:636)
Caused by: com.my.application.IOException: null
    at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1692)
    at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)
    at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
    at org.apache.abcd2.WriteTo(SOAPMessageFormatter.java:79)
    ... 27 more

所以现在如果我想得到像LOGLEVEL类名这样的值，并且由于我得到了loglevel和类名的值而没有得到引起的消息那么......怎么可能？

以下是我的配置文件。

input{
    file{
         path => "D:\Log\application.log"
         start_position => beginning

         codec => multiline{
         pattern => "%{TIMESTAMP_ISO8601}"
         what => "next"
         negate => true
        }
    }
}
filter{

        grok{

            match => ["message","^%{TIMESTAMP_ISO8601}<%{LOGLEVEL}><(?<JavaClass>.*[:].*)>"]
        }
        mutate {
        gsub => ['message', "\n", ""]
        gsub => ['message', "\t", ""]
        }
}
output {
        stdout { }
        elasticsearch {
            index => "ABCD_%{+YYYY.MM.dd}"

    }
}

我主要关注的是解析timestamp loglevel classname和由值引起的

Answer 1

您的多行编解码器错误，这就是您的配置无效的原因。（我测试过了）。

当我在我的盒子上使用它时，这是你的配置的stdout（你没有发布）：

artur@pandaadb:~/dev/logstash$ ./logstash-2.3.2/bin/logstash -f conf2/
Settings: Default pipeline workers: 8
Pipeline main started
{
    "@timestamp" => "2016-06-22T09:19:01.896Z",
       "message" => "2016-06-02 17:00:32<ERROR><CommonsHTTPTransportSender:361>: ",
      "@version" => "1",
          "path" => "/home/artur/tmp/logstash/in2/test.log",
          "host" => "pandaadb",
     "JavaClass" => "CommonsHTTPTransportSender:361"
}

查看您的消息甚至没有任何信息？没有什么可以匹配的，因为你的mutliline不起作用。我希望这首先是一个问题：

您的模式不反映匹配行的开头
当你否定模式时，你应该做一个“前一个”而不是下一个。

所以这是一个工作配置（我使用的是多行过滤器而不是编解码器）：

multiline {
                pattern => "^%{TIMESTAMP_ISO8601}"
                negate => true
                what => "previous"
        }

有了这个，我可以更新你的grok以接受原因：

grok {
        match => ["message","^%{TIMESTAMP_ISO8601:ts}<%{LOGLEVEL:log}><(?<JavaClass>.*[:].*)>.*Caused by:%{GREEDYDATA:data}"]
    }

在我的盒子上输入你的输入后，我得到：

artur@pandaadb:~/dev/logstash$ ./logstash-2.3.2/bin/logstash -f conf2/
Settings: Default pipeline workers: 8
Defaulting pipeline worker threads to 1 because there are some filters that might not work with multiple worker threads {:count_was=>8, :filters=>["multiline"], :level=>:warn}
Pipeline main started
{
       "message" => "2016-06-02 17:00:32<ERROR><CommonsHTTPTransportSender:361>: \norg.apache.abcd2.abcdFault\n    at org.apache.abcd2.abcdFault.makeFault(abcdFault.java:430)\n    at org.apache.abcd2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:83)\n    at org.apache.abcd2.transport.http.CommonsHTTPTransportSender.sendUsingOutputStream(CommonsHTTPTransportSender.java:358)\n    at java.lang.Thread.run(Thread.java:636)\nCaused by: com.my.application.IOException: null\n    at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1692)\n    at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)\n    at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)\n    at org.apache.abcd2.WriteTo(SOAPMessageFormatter.java:79)\n    ... 27 more",
      "@version" => "1",
    "@timestamp" => "2016-06-22T09:22:38.227Z",
          "path" => "/home/artur/tmp/logstash/in2/test.log",
          "host" => "pandaadb",
          "tags" => [
        [0] "multiline"
    ],
            "ts" => "2016-06-02 17:00:32",
           "log" => "ERROR",
     "JavaClass" => "CommonsHTTPTransportSender:361",
          "data" => " com.my.application.IOException: null\n    at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1692)\n    at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)\n    at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)\n    at org.apache.abcd2.WriteTo(SOAPMessageFormatter.java:79)\n    ... 27 more"
}

希望有所帮助。

为了将来参考，关于logstash问题，有一个示例将stdin和stdout打印到stdout总是有帮助的，因为它可以很快地再现。

stdout（rubydebug）的输出也会告诉你原始消息的确切内容，并且很容易看出多行不起作用，这也是导致问题的原因。

干杯！

阿图尔

是否有可能在Logstash confilg中解析由java日志引起的？

1 个答案: