Question

我正在尝试从日志文件中解析多行数据。我尝试过多行编解码器和多行过滤器。但它对我不起作用。

记录数据

INFO 2014-06-26 12:34:42,881 [4] [HandleScheduleRequests] Request Entity:
User Name : user
DLR : 04
Text : string
Interface Type : 1
Sender : sdr
DEBUG 2014-06-26 12:34:43,381 [4] [HandleScheduleRequests] Entitis is : 1 System.Exception

这是配置文件

input {

   file {
           type => "cs-bulk"
           path =>
           [
                   "/logs/bulk/*.*"
           ]

           start_position => "beginning"
           sincedb_path => "/logstash-1.4.1/bulk.sincedb"

           codec => multiline {
                    pattern => "^%{LEVEL4NET}"
                    what => "previous"
                    negate => true
           }
   }
}

output {

  stdout { codec => rubydebug }

  if [type] == "cs-bulk" {

     elasticsearch {
       host => localhost
       index => "cs-bulk"
     }
   }
}

filter {

  if [type] == "cs-bulk" {
    grok {
        match => { "message" => "%{LEVEL4NET:level} %{TIMESTAMP_ISO8601:time} %{THREAD:thread} %{LOGGER:method} %{MESSAGE:message}" }
         overwrite => ["message"]
    }
  }
}

这是我在logstash解析多线部分时得到的它只是得到第一行，并将其标记为多行。 未解析的其他行！

{
    "@timestamp" => "2014-06-27T16:27:21.678Z",
       "message" => "Request Entity:",
      "@version" => "1",
          "tags" => [
        [0] "multiline"
    ],
          "type" => "cs-bulk",
          "host" => "lab",
          "path" => "/logs/bulk/22.log",
         "level" => "INFO",
          "time" => "2014-06-26 12:34:42,881",
        "thread" => "[4]",
        "method" => "[HandleScheduleRequests]"
}

Answer 1

在您的grok模式的开头放置(?m)。这将使正则表达式不会停留在\n。

Answer 2

不太确定会发生什么，但使用多线过滤器而不是像这样的编解码器：

input {
  stdin {
  }
}

filter {
     multiline {
                    pattern => "^(WARN|DEBUG|ERROR)"
                    what => "previous"
                    negate => true
           }
}

在我的测试中有效吗？

{
       "message" => "INFO 2014-06-26 12:34:42,881 [4] [HandleScheduleRequests] Request Entity:\nUser Name : user\nDLR : 04\nText : string\nInterface Type : 1\nSender : sdr",
      "@version" => "1",
    "@timestamp" => "2014-06-27T20:32:05.288Z",
          "host" => "HOSTNAME",
          "tags" => [
        [0] "multiline"
    ]
}
{
       "message" => "DEBUG 2014-06-26 12:34:43,381 [4] [HandleScheduleRequests] Entitis is : 1 System.Exception",
      "@version" => "1",
    "@timestamp" => "2014-06-27T20:32:05.290Z",
          "host" => "HOSTNAME"
}

除了......我用过的测试文件从不打印出最后一行（因为它还在寻找更多信息）

Logstash 1.4.1多行编解码器无法正常工作

2 个答案: