Question

我尝试使用ElasticSearch将一些数据从文件索引到Logstash。

如果我不使用Date filter来替换@timestamp一切正常，但在使用过滤器时我没有获得所有数据

我无法弄清楚为什么Logstash command line中的Elasticsearch和@timestamp之间存在差异。

Logstash conf

filter { 
    mutate {
        replace => {
            "type" => "dashboard_a"
        }
    }
    grok {
        match => [ "message", "%{DATESTAMP:Logdate} \[%{WORD:Severity}\] %{JAVACLASS:Class} %{GREEDYDATA:Stack}" ]
    }
    date {                        
        match => [ "Logdate", "dd-MM-yyyy hh:mm:ss,SSS" ]
    }   
}

Logstash命令行跟踪

{
**"@timestamp" => "2014-08-26T08:16:18.021Z",**
   "message" => "26-08-2014 11:16:18,021 [DEBUG] com.fnx.snapshot.mdb.SnapshotMDB  - SnapshotMDB Ctor is called\r",
  "@version" => "1",
      "host" => "bts10d1",
      "path" => "D:\\ElasticSearch\\logstash-1.4.2\\Dashboard_A\\Log_1\\6.log",
      "type" => "dashboard_a",
   "Logdate" => "26-08-2014 11:16:18,021",
  "Severity" => "DEBUG",
     "Class" => "com.fnx.snapshot.mdb.SnapshotMDB",
     "Stack" => " - SnapshotMDB Ctor is called\r"
  }

ElasticSearch结果

{
    "_index": "logstash-2014.08.28",
    "_type": "dashboard_a",
    "_id": "-y23oNeLQs2mMbyz6oRyew",
    "_score": 1,
    "_source": {
        **"@timestamp": "2014-08-28T14:31:38.753Z",
        **"message": "15:07,565 [DEBUG] com.fnx.snapshot.mdb.SnapshotMDB  - SnapshotMDB Ctor is called\r",
        "@version": "1",
        "host": "bts10d1",
        "path": "D:\\ElasticSearch\\logstash-1.4.2\\Dashboard_A\\Log_1\\6.log",
        "type": "dashboard_a",
        "tags": ["_grokparsefailure"]
    }
}

Answer 1

请确保所有日志都是格式化的！

您可以在logstash命令行跟踪中看到日志

26-08-2014 11：16：18,021 [DEBUG] com.fnx.snapshot.mdb.SnapshotMDB - SnapshotMDB Ctor被称为\ r

但是，在elastsicsearch中，日志是

15：07,565 [DEBUG] com.fnx.snapshot.mdb.SnapshotMDB - SnapshotMDB Ctor被称为\ r＆＃34;，

两个日志有不同的时间，格式不一样！第二个没有关于白天的任何信息，因此它将导致grok filter解析错误。您可以去检查原始日志。或者你可以提供原始日志样本，如果它们都是格式的话，可以进行更多讨论！

Logstash和ElasticSearch过滤日期@timestamp问题

1 个答案: