Question

我在服务器上运行ELK Stack，我在其中集成了Squid access.log文件（组合格式也可以获取用户代理）。它在Kibana中正确显示所有日志，我的logstash.stdout文件正确显示条目：

{
    "message" => "192.168.2.26 - - [06/Apr/2016:19:08:51 +0200] \"GET https://r2---sn-4gxx-25gs.googlevideo.com/videoplayback/id/o-ABKSIYjx-CBdsfnjHVg6926-k9evTCAai6SfBJ7AGoNA/itag/311/source/youtube/govp/slices%3D0-713,52432612-53212629/gosq/63/file/seg.ts?dnc=1&cpn=yH1AZUBmkDfs_UGh&sgovp=clen=186211243;dur=1154.849;itag=298;lmt=1459931359321727;gir=yes&ip=88.177.83.67&ratebypass=yes&mt=1459962265&playlist_type=DVR&ms=au&ipbits=0&pl=12&mv=m&mm=31&expire=1459984025&upn=egKraIHQnU8&pfa=1&keepalive=yes&fexp=9408541,9416137,9416891,9417405,9420452,9422542,9422596,9426927,9427015,9427902,9428269,9428333,9428398,9428567,9428769,9430014,9430058,9431045,9431614,9431850,9431976,9432026,9432429,9432631,9433548&sver=3&sparams=hls_chunk_host,id,initcwndbps,ip,ipbits,itag,mm,mn,ms,mv,pfa,pl,playlist_type,ratebypass,requiressl,sgovp,source,expire&signature=C55E2BDA79965F9062CB2306C7B082776587104A.3BACE1898A53453B99CF50CA230E127E8EC095E8&requiressl=yes&initcwndbps=682500&hls_chunk_host=r2---sn-4gxx-25gs.googlevideo.com&mn=sn-4gxx-25gs&key=yt6&dur=5.067 HTTP/1.1\" 200 350062 \"-\" \"AppleCoreMedia/1.0.0.13B143 (iPad; U; CPU OS 9_1 like Mac OS X; fr_fr)\" TCP_MISS_ABORTED:ORIGINAL_DST",
   "@version" => "1",
 "@timestamp" => "2016-04-06T17:08:51.000Z",
       "path" => "/var/log/squid/access.log",
       "host" => "secbox.localdomain.local",
       "type" => "squid",
   "clientip" => "192.168.2.26",
      "ident" => "-",
       "auth" => "-",
  "timestamp" => "06/Apr/2016:19:08:51 +0200",
       "verb" => "GET",
    "request" => "https://r2---sn-4gxx-25gs.googlevideo.com/videoplayback/id/o-ABKSIYjx-CBdsfnjHVg6926-k9evTCAai6SfBJ7AGoNA/itag/311/source/youtube/govp/slices%3D0-713,52432612-53212629/gosq/63/file/seg.ts?dnc=1&cpn=yH1AZUBmkDfs_UGh&sgovp=clen=186211243;dur=1154.849;itag=298;lmt=1459931359321727;gir=yes&ip=88.177.83.67&ratebypass=yes&mt=1459962265&playlist_type=DVR&ms=au&ipbits=0&pl=12&mv=m&mm=31&expire=1459984025&upn=egKraIHQnU8&pfa=1&keepalive=yes&fexp=9408541,9416137,9416891,9417405,9420452,9422542,9422596,9426927,9427015,9427902,9428269,9428333,9428398,9428567,9428769,9430014,9430058,9431045,9431614,9431850,9431976,9432026,9432429,9432631,9433548&sver=3&sparams=hls_chunk_host,id,initcwndbps,ip,ipbits,itag,mm,mn,ms,mv,pfa,pl,playlist_type,ratebypass,requiressl,sgovp,source,expire&signature=C55E2BDA79965F9062CB2306C7B082776587104A.3BACE1898A53453B99CF50CA230E127E8EC095E8&requiressl=yes&initcwndbps=682500&hls_chunk_host=r2---sn-4gxx-25gs.googlevideo.com&mn=sn-4gxx-25gs&key=yt6&dur=5.067",
"httpversion" => "1.1",
   "response" => "200",
      "bytes" => "350062",
   "referrer" => "\"-\"",
      "agent" => "\"AppleCoreMedia/1.0.0.13B143 (iPad; U; CPU OS 9_1 like Mac OS X; fr_fr)\"",
       "tags" => [
    [0] "squid"
]

}

但是，logstash.log文件声明了日期错误：

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"06/Apr/2016:18:51:04 +0200\" is malformed at \"/Apr/2016:18:51:04 +0200\""}}}}, :level=>:warn}

我的/etc/logstash/conf.d/squid.conf文件定义如下：

input {
  file {
    type => "squid"
    path => [ "/var/log/squid/access.log" ]
  }
}

filter {
  if [type] == "squid" {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}" }
        add_tag => ["squid"]
    }
    date {
        match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  }
}

output {
  elasticsearch { hosts => localhost }
  stdout { codec => rubydebug }
}

我不明白为什么我这个错误消息，而日志似乎被logstash正确解析。

有什么想法吗？提前感谢您的意见。

Answer 1

虽然已经很长时间了，但是认为答案会帮助那些面临这个问题的人。 Logstash date invalid format

url给了我答案，使用mutate替换时间格式变异{ gsub =＆gt; [＆＃34;时间戳＆＃34;＆＃34; /＆＃34;＆＃34; - ＆＃34;] } 日期{ match =＆gt; [＆＃34;时间戳＆＃34;，＆＃34;您的格式＆＃34;] }

Logstash Squid日期无效格式

1 个答案: