Logstash rsyslog的多行模式

时间:2018-10-01 07:46:28

标签: elasticsearch logstash multiline syslog rsyslog

我确实通过多行编解码器从rsyslog(systemd)解析了一条消息,但是它无法正常工作。

我使用这种模式

pattern => "^%{SYSLOGBASE} %{TIMESTAMP_ISO8601}"

解析消息的开头:

Sep 28 14:42:10 ibcore9t tarantool[148700]: 2018-09-28 14:42:10.805

完整的消息是这样的:

Oct 01 10:27:21 ibcore9t tarantool[148700]: 2018-10-01 10:27:21.533 [148700] main/14364/http/172.25.8.18:64209 init.lua:179 E> Pipeline execution failed: Function "focus_spotforward_couponpayment_map" in pipeline "focus_spotforward_couponpayment_handle": ./ib-common/sandbox.lua:130: Trade not found
Oct 01 10:27:21 ibcore9t tarantool[148700]: stack traceback:
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./ib-common/pipeline/init.lua:37: in function 'call_function'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./ib-common/pipeline/init.lua:75: in function 'call_pipeline'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./ib-common/pipeline/init.lua:111: in function 
Oct 01 10:27:21 ibcore9t tarantool[148700]: [C]: in function 'pcall'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./ib-common/pipeline/init.lua:109: in function 'call'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./ib-core/server.lua:119: in function 
Oct 01 10:27:21 ibcore9t tarantool[148700]: [C]: in function 'xpcall'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./ib-common/errors.lua:93: in function 'pcall'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./ib-core/server.lua:162: in function 
Oct 01 10:27:21 ibcore9t tarantool[148700]: [C]: in function 'xpcall'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ...
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./ib-common/errors.lua:93: in function 'pcall'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./ib-common/funcall.lua:41: in function 'call'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./t-connect/soapserver.lua:139: in function 'callback'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ./ib-common/http.lua:201: in function 'sub'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ...e/tarantool/core0/.rocks/share/tarantool/http/server.lua:610: in function 
Oct 01 10:27:21 ibcore9t tarantool[148700]: [C]: in function 'pcall'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ...e/tarantool/core0/.rocks/share/tarantool/http/server.lua:687: in function 'process_client'
Oct 01 10:27:21 ibcore9t tarantool[148700]: ...e/tarantool/core0/.rocks/share/tarantool/http/server.lua:1132: in function 
Oct 01 10:27:21 ibcore9t tarantool[148700]: [C]: in function 'pcall'
Oct 01 10:27:21 ibcore9t tarantool[148700]: builtin/socket.lua:1059: in function 

但是我只看到基巴纳语的第一行

enter image description here

Pipeline execution failed: Function "focus_spotforward_couponpayment_map" in pipeline "focus_spotforward_couponpayment_handle": ./ib-common/sandbox.lua:130: Trade not found

我的logstash.conf:

input {
    udp {
    port => 5000
    type => syslog
    codec => multiline {
    pattern => "^%{SYSLOGBASE} %{TIMESTAMP_ISO8601}"
    negate => true
    what => "next"
     }
    }
}

filter {
    grok {
      match => [ "message", "%{SYSLOGBASE} %{TIMESTAMP_ISO8601:timestamp_tarantool} \[%{NUMBER:id}\] %{WORD}/%{NUMBER}/%{WORD:method} %{WORD:log_level}> %{GREEDYDATA:msg}",
                 "message", "%{SYSLOGBASE} %{TIMESTAMP_ISO8601:timestamp_tarantool} \[%{NUMBER:id}\] %{WORD}/%{NUMBER}/%{WORD:method}/%{IP}:%{NUMBER} %{WORD:log_level}> %{GREEDYDATA:msg}",
                 "message", "%{SYSLOGBASE} %{TIMESTAMP_ISO8601:timestamp_tarantool} \[%{NUMBER:id}\] %{WORD}/%{NUMBER}/%{WORD:method}/%{IP}:%{NUMBER} %{WORD}.%{WORD}:%{NUMBER} %{WORD:log_level}> %{GREEDYDATA:msg}"
               ]
       }

     mutate {
      replace => { "message" => "%{msg}" }
      remove_field => [ "msg" ]
     }

     if "_grokparsefailure" in [tags] { drop {} }

     if [message] =~ "queue" {
      mutate {
       add_field => { "repair_queue" => "%{message}"}
        }
       }

}

output {
    elasticsearch {
    action => "index"
    index => "logstash-%{+YYYY.MM.dd}"
    hosts  => "ibmetrics2:9200"
 }
}

请帮助

0 个答案:

没有答案