Logstash Grok和KV

Question

我无法让KV过滤器在logstash中工作，也无法弄清楚我做错了什么。非常感谢任何帮助：）

配置

filter {
if [type] == "syslog" and [message] =~ "SFDC-IT"{
grok {
  match => { "message" => "%{DATE:date} %{TIME:time}%{GREEDYDATA:sfdc_message}"}
}
kv {
source => "sfdc_message"
include_keys => ["Application Information","Process ID","Application Name","Inbound Source Address","Source Port","Destination Address","Destination Port","Protocol","Filter Run-Time ID","Layer Name","Layer Run-Time ID"]
field_split => " "
value_split => ":"
target => "data"
remove_field => [ "sfdc_message" ]
    }
  }
}

来自rubydebug的输出（我试图使用kv过滤器拆分sfdc​​_message）

         "message" => "<133>RealSource: \"dc-wp1\" Environment: \"SFDC-IT\" UUID: \"\" RawMsg: EvntSLog: Security`2015-12-12 23:52:45`Microsoft-Windows-Security-Auditing`Information`5156`Audit Success`[AUS]`N\\A`4`0x8020000000000000`12810`2147483647`The Windows Filtering Platform has permitted a connection. Application Information: Process ID: 876 Application Name: \\device\\harddiskvolume2\\windows\\system32\\svchost.exe Network Information: Direction: Inbound Source Address: 10.1.34.2 Source Port: 49153 Destination Address: 10.1.45.199 Destination Port: 50278 Protocol: 6 Filter Information: Filter Run-Time ID: 67090 Layer Name: Receive/Accept Layer Run-Time ID: 44",
    "@version" => "1",
  "@timestamp" => "2015-12-13T00:01:37.061Z",
        "type" => "syslog",
        "host" => "10.1.45.199",
        "date" => "2015-12-12",
        "time" => "23:52:45",
"sfdc_message" => "`Microsoft-Windows-Security-Auditing`Information`5156`Audit Success`[AUS]`N\\A`4`0x8020000000000000`12810`2147483647`The Windows Filtering Platform has permitted a connection. Application Information: Process ID: 876 Application Name: \\device\\harddiskvolume2\\windows\\system32\\svchost.exe Network Information: Direction: Inbound Source Address: 10.1.34.2 Source Port: 49153 Destination Address: 10.1.45.199 Destination Port: 50278 Protocol: 6 Filter Information: Filter Run-Time ID: 67090 Layer Name: Receive/Accept Layer Run-Time ID: 44"

Answer 1

我能够在不调用grok的情况下使用它。

input {
  tcp {
    port => 514
    type => syslog
    codec => plain {
         charset => "ISO-8859-1"
    }
  }
  udp {
port => 514
type => syslog
codec => plain {
         charset => "ISO-8859-1"
}
  }
}
filter
{
  if [message] =~ /^<181>/
{
kv {
type => syslog
add_field => { "log_type" => "CISE" }
remove_field => [ "Step", "cisco-av-pair", "NetworkDeviceGroups", "message"]
   }
}

  else if [message] =~ /^<133>/
{
mutate {
gsub => [ "message", "\"", ""]
gsub => [ "message", ": ", "="]
gsub => [ "message", "Inbound Source Address", "Inbound_Source_Address"]
gsub => [ "message", "Source Port", "Source_Port"]
gsub => [ "message", "Destination Address", "Destination_Address"]
gsub => [ "message", "Destination Port", "Destination_Port"]
gsub => [ "message", "Layer Name", "Layer_Name"]
gsub => [ "message", "Application Name", "Application_Name"]
gsub => [ "message", "Information=Direction=", ""]
       }
kv {
type => syslog
add_field => { "log_type" => "AD-133" }
remove_field => [ "RawMsg", "Information", "ID", "message"]
   }
}
  else if [message] =~ /^<134>/
{
mutate {
gsub => [ "message", "\"", ""]
gsub => [ "message", ": ", "="]
       }
kv {
type => syslog
add_field => { "log_type" => "AD-134" }
   }
}
}

output {
file {
codec => "rubydebug"
path => ["/tmp/logstash-out.out"]
}
  if [type] == "syslog" and "_grokparsefailure" in [tags] {
    file { path => "/var/log/failed_syslog_events-%{+YYYY-MM-dd}" }
}
  stdout { codec => rubydebug }
}