elasticsearch filebeat

时间:2018-03-17 15:40:34

标签: logstash logstash-grok filebeat

我在@timestamp字段中为elasticsearch / filebeat获取了错误的值。 我的文件管道定义

    curl -H 'Content-Type: application/json' -XPUT "logger:9200/_ingest/pipeline/app_log" -d'
    {
      "description" : "Ingest pipeline for Jetty server log",
      "processors" : [
        {
          "grok": {
            "field": "message",
            "patterns": ["%{TIMESTAMP_ISO8601:timestamp} (%{UUID:accessid})? \\[(?<threadname>[^\\]]+)\\] %{LOGLEVEL:level} %{DATA:classname} - %{GREEDYDATA:message}"]
          }
        },
        {
          "date": {
            "field": "timestamp",
            "formats": [ "yyyy-mm-dd H:m:s,SSS" ]
          }
        }
      ],
      "on_failure" : [{
        "set" : {
          "field" : "error.message",
          "value" : "{{ _ingest.on_failure_message }}"
        }
      }]
    }'

某些样本记录器行的模拟结果。 (使用logback配置)

curl -H 'Content-Type: application/json' -XPOST "logger:9200/_ingest/pipeline/app_log/_simulate?pretty" -d'
{
  "docs": [
    {
      "_source": {
        "message": "2018-03-17 22:38:39,079 bab3157d-a11c-4dba-a6d6-c47ae0de2b7f [qtp224100622-174782] INFO  i.n.core.services.cache.CacheBuilder - Key : ChIJrTTTJkdsrjsRXkrYRKRRfd8-seo-localitiesv1 is returned from cache"

      }
    },
    {
      "_source": {
        "message": "2017-12-12 01:14:12,079  [qtp224100622-185269] WARN  i.n.m.cache.sdk.RedisCacheProvider - No matching policy: class in.nobroker.core.domain.Token"
      }
    }
  ]
}' 

此模拟的结果:

  

{“docs”:[       {         “doc”:{           “_index”:“_ index”,           “_type”:“_ type”,           “_id”:“_ id”,           “_资源” : {             “accessid”:“bab3157d-a11c-4dba-a6d6-c47ae0de2b7f”,             “@timestamp”:“2018-01-17T22:38:39.079Z”,             “classname”:“i.n.core.services.cache.CacheBuilder”,             “级别”:“信息”,             “message”:“密钥:从缓存中返回ChIJrTTTJkdsrjsRXkrYRKRRfd8-seo-localitiesv1”,             “timestamp”:“2018-03-17 22:38:39,079”,             “threadname”:“qtp224100622-174782”           },           “_ingest”:{             “timestamp”:“2018-03-17T15:35:35.543Z”           }         }       },       {         “doc”:{           “_index”:“_ index”,           “_type”:“_ type”,           “_id”:“_ id”,           “_资源” : {             “@timestamp”:“2017-01-12T01:14:12.079Z”,             “classname”:“i.n.m.cache.sdk.RedisCacheProvider”,             “等级”:“警告”,             “message”:“没有匹配的策略:class in.nobroker.core.domain.Token”,             “timestamp”:“2017-12-12 01:14:12,079”,             “threadname”:“qtp224100622-185269”           },           “_ingest”:{             “timestamp”:“2018-03-17T15:35:35.543Z”           }         }       }]}

请注意@timestamp字段与timestamp字段完全不同。

2 个答案:

答案 0 :(得分:0)

似乎您提供了错误的日期格式

Error:(27, 0) Could not find method buildTypes() for arguments [build_4fb364oy5uo8h1irwj22vkw4h$_run_closure2@601eabab] on project ':app' of type org.gradle.api.Project.
<a href="openFile:G:\my game of now\zombies\android studio\app\build.gradle">Open File</a>

由于doc显示&#34; mm:两位数分钟,如果需要,则填充零填充&#34;而不是你期望的那个月。 尝试

{
  "date": {
    "field": "timestamp",
    "formats": [ "yyyy-mm-dd H:m:s,SSS" ]
  }
}

答案 1 :(得分:0)

Filebeat没有日期处理器。 Elasticsearch具有处理器。要将日志中的时间戳记用作filebeat中的@timestamp,请使用Elasticsearch中的摄取管道。