Question

有人可以告诉我我做错了什么，或者Logstash为什么不想解析ISO8601时间戳？

我收到的错误消息是

操作失败...＆＃34;错误＆＃34; =＆gt; {＆＃34;输入＆＃34; =＆gt;＆＃34; mapper_parsing_exception＆＃34;，＆＃34;原因＆＃34; =＆gt;＆＃34;无法解析[timestamp]＆＃34;，＆＃34; caused_by＆＃34; =＆gt; {＆＃34;输入＆＃34; =＆gt;＆＃34; illegal_argument_exception＆＃34;，＆＃34;原因＆＃34; =＆gt;＆＃34;无效格式：\＆＃34; 2017-03-24 12:14:50 \＆＃34;是在＃03; 17-03-24 12时14分50秒\＆＃34;＆＃34;}}

示例日志文件行（IP地址中的最后一个字节故意替换为000）

2017-03-24 12:14:50 87.123.123.000 12345678.domain.com GET /smil:stream_17.smil/chunk_ctvideo_ridp0va0r600115_cs211711500_mpd.m4s - HTTP/1.1 200 750584 0.714 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" https://referrer.domain.com/video/2107 https fra1 "HIT, MISS" 12345678.domain.com

GROK模式（使用http://grokconstructor.appspot.com/do/match验证）

RAW %{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{IPV4:clientip}%{SPACE}%{HOSTNAME:http_host}%{SPACE}%{WORD:verb}%{SPACE}\/(.*:)?%{WORD:stream}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:protocol}\/%{NUMBER:httpversion}%{SPACE}%{NUMBER:response}%{SPACE}%{NUMBER:bytes}%{SPACE}%{SECOND:request_time}%{SPACE}%{QUOTEDSTRING:agent}%{SPACE}%{URI:referrer}%{SPACE}%{WORD}%{SPACE}%{WORD:location}%{SPACE}%{QUOTEDSTRING:cache_status}%{SPACE}%{WORD:account}%{GREEDYDATA}

Logstash配置（输入端）：

input {
    file {
      path => "/subfolder/logs/*"
      type => "access_logs"
      start_position => "beginning"
    }
}
filter {
    # skip first two lines in log file with comments
    if [message] =~ /^#/ {
        drop { }
    }

    grok {
        patterns_dir => ["/opt/logstash/patterns"]
        match => { "message" => "%{RAW}" }
    }

    date {
        match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
        locale => "en"
    }

    # ... (rest of the config omitted for readability)
}

Answer 1

所以我很确定这是由于timestamp字段映射到Elasticsearch中的一个类型而导致的，它不能解析。如果您发布索引映射，我很乐意看看它。

注意：您可以通过添加remove_field来快速解决此问题，因为如果date过滤器成功，该字段的值将被拉入@timestamp。现在，您在两个字段中存储了相同的值。然后您不必担心该字段的映射。 :)

date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
    locale => "en"
    remove_field => [ "timestamp" ]
}

Logstash解析器错误，时间戳格式错误

1 个答案: