Question

我有来自NGINX的以下日志：

111.111.111.111,11.11.11.11 - 11.11.11.11 [06 / May / 2016：08：26：10 +0000]“POST / some-service / GetSomething HTTP / 1.1”499 0“ - ”“Jakarta Commons- HttpClient / 3.1“”7979798797979799“ 59.370 - ”{\ x0A \ x22correlationId \ x22：\ x22TestCorr1 \ x22 \ x0A}“

Logstash将是这样的：

input { stdin {} }
output { stdout { codec => "rubydebug" } }

filter {
   grok {
 match => { "message" => "%{COMBINEDAPACHELOG} %{QS:partner_id} %{NUMBER:req_time} %{GREEDYDATA:extra_fields}" }
 add_field => [ "received_at", "%{@timestamp}" ]
 add_field => [ "received_from", "%{host}" ]
  }


mutate {
 gsub => ["extra_fields", "\"","",
      "extra_fields", "\\x0A","",
      "extra_fields", "\\x22",'\"',
      "extra_fields", "(\\)",""
  ]
}

json {
  source => "extra_fields"
  target => "extra_fields_json"
}


mutate {
  add_field => {
    "correlationId" => "%{[extra_fields_json][correlationId]}"
  }
}

}

问题是 req_time 是字符串，所以我需要使用以下模板转换为float：

{
  "template" : "filebeat*",
 "settings" : {
   "index.refresh_interval" : "5s"
 },

 "mappings" : {
  "properties" : {
    "@timestamp": { "type": "date" },
    "partner_id": { "type": "string", "index": "not_analyzed" },
    "@version": { "type": "string", "index": "not_analyzed" },
    "req_time" : { "type" : "float", "index" : "not_analyzed" }, 
    "res_time" : { "type" : "string", "index" : "not_analyzed" },
    "purchaseTime" : { "type" : "date", "index" : "not_analyzed" },
    "received_at" : { "type" : "date", "index" : "not_analyzed" },
    "itemPrice" : { "type" : "double", "index" : "not_analyzed" },
    "total" : { "type" : "integer", "index" : "not_analyzed" },
    "bytes" : { "type" : "double", "index" : "not_analyzed" }
  }
   }
 }

}

验证使用： curl -XGET'http://localhost:9200/filebeat-2016.06.30/_mapping/field/req_time'

我得到了：

{“filebeat-2016.06.30”：{“mappings”：{“nginxlog”：{“req_time”：{“full_name”：“req_time”，“mapping”：{“req_time”：{“输入“：”string“} }}}}}}

所以我的模板肯定不起作用。有人可以帮忙吗？

Answer 1

最后，我刚删除了模板，让ES guest为字段类型。它确实有效。

NGINX自定义日志

1 个答案: