Question

我正在尝试从logstash输出中获取所需的时间戳格式。如果我在syslog中使用这种格式

，我就无法得到

请分享您对转换为_source字段中其他格式的想法，例如Yyyy-mm-ddThh：mm：ss.sssZ格式？

filter {
  grok {
        match => [ "logdate", "Yyyy-mm-ddThh:mm:ss.sssZ" ]
       overwrite => ["host", "message"]
  }

_source: {
message: "activity_log: {"created_at":1421114642210,"actor_ip":"192.168.1.1","note":"From system","user":"4561c9d7aaa9705a25f66d","user_id":null,"actor":"4561c9d7aaa9705a25f66d","actor_id":null,"org_id":null,"action":"user.failed_login","data":{"transaction_id":"d6768c473e366594","name":"user.failed_login","timing":{"start":1422127860691,"end":14288720480691,"duration":0.00257},"actor_locatio

我在syslog文件中使用此代码

filter {
  if [message] =~ /^activity_log: / {
    grok {
      match => ["message", "^activity_log: %{GREEDYDATA:json_message}"]
    }
    json {
      source => "json_message"
      remove_field => "json_message"
    }
    date {
      match => ["created_at", "UNIX_MS"]
    }
    mutate {
      rename => ["[json][repo]", "repo"]
      remove_field => "json"
    }
  }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

感谢

"message" => "<134>feb  1 20:06:12 {\"created_at\":1422765535789, pid=5450 tid=28643 version=b0b45ac proto=http  ip=192.168.1.1 duration_ms=0.165809 fs_sent=0 fs_recv=0 client_recv=386 client_sent=0 log_level=INFO msg=\"http op done: (401)\" code=401" }
      "@version" => "1",
    "@timestamp" => "2015-02-01T20:06:12.726Z",
          "type" => "activity_log",
          "host" => "192.168.1.1"

Answer 1

grok filter中的模式没有意义。你正在使用Joda-Time模式（通常用于date filter）而不是grok模式。

您的message字段似乎包含JSON对象。这很好，因为它很容易解析。将“activity_log：”之后的部分提取到临时json_message字段

grok {
  match => ["message", "^activity_log: %{GREEDYDATA:json_message}"]
}

并使用json filter将该字段解析为JSON（如果操作成功，则删除临时字段）：

json {
  source => "json_message"
  remove_field => ["json_message"]
}

现在，您应该拥有邮件顶层原始邮件字段中的字段，包括带有您要提取的时间戳的created_at字段。该数字是自纪元以来的毫秒数，因此您可以使用date filter中的UNIX_MS模式将其提取到@timestamp：

date {
  match => ["created_at", "UNIX_MS"]
}

logstash输出未显示所需的时间戳

1 个答案: