Question

我使用Logstach和logstach转发器来提取我的日志文件，我有一些包含JSON格式的日志文件：

[2015-11-05 17:39:22.200] [INFO] dashboard - request :{ "user": "admin", "headers": {"host":"localhost:0000","connection":"keep-alive","accept":"application/json, text/plain, */*","user-agent":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36",
"referer":"http://localhost:0000/","accept-encoding":"gzip, deflate, sdch","accept-language":"fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4","cookie":"splunkweb_csrf_token_8000=0000000000000000; _ga=GA1.1.0000000.1445436724; connect.sid=s%3AmlK46TZsFa202R5o3nwuHTbmHjehmJiO.JxlNGOXWuY%2Fp0RenTWsxLLDZvVgt8aVQ%2FpKryJsGCpw"},
"method": "GET", "url" : "/count_event", "params" :{"_type":"twitter,facebook,forum","year":"2013,2014,2015","month":"January,February,March,April,May,June,July,August,September,October,November,December"}}

所以，我需要在ES中保存这些信息：

时间戳：2015-11-05 17：39：22.200 类型：INFO msg：JSON中的所有信息

这是我的logstach文件配置：

input {
   lumberjack {
      port => 5043
      type => "logs"
      ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
      ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
   }
}
filter {
    grok {
      patterns_dir => ["/home/logstash/logstach-2.0.0/pattern"]
      add_tag => [ "valid" ]
      match => { "message" => "^\[%{TIMESTAMP_ISO8601:timestamp}\]\[%{DATA:typemessage}\] %{DATA:appname} \- %{GREEDYDATA:msg}}
    }

   date {
     match => ["timestamp", "YYYY-MM-dd HH:mm:ss,SSS"]
     remove_field => ["timestamp"]
   }

    if "valid" not in [tags] {
      drop { }
    }
}
output {
   elasticsearch {
         hosts =>"192.168.1.153:9200"
         index =>"logs"
   }
   stdout { codec => rubydebug }
}

使用此配置，我在字段msg上获得了字符串格式。

Answer 1

您需要告诉logstash将字段解析为json。 json{} filter用于此目的。给它你的'msg'字段作为输入。

但是请注意，您的原始grok应该在'msg'字段中包含输入的“request：”部分，这是无效的json。你需要调整你的grok模式，只在你发送到json过滤器的字段中放入有效的json。

从日志Logstach

1 个答案: