Logstash中的日志无法正确解析

时间:2019-11-11 09:54:50

标签: elasticsearch logstash kibana logstash-grok filebeat

我的体系结构是

  

Filebeat A (远程)> Logstash A(2条管道)> Elasticsearch A> Kibana A

     

Filebeat B (远程)> Logstash A(2条管道)> Elasticsearch A> Kibana A

用于日志分析。

说我的日志格式为abc_logs-yyyy.mm.dd.log

我的Filebeats将日志推送到Logstash(我可以在数据/注册表文件中看到),但是Logstash 正在创建索引用于某些日志文件。

例如,abc_logs-2019.11.02.log在我的日志位置中,并且还Filebeat将其推送到Logstash。但是我Elasticsearch中看不到任何创建的索引

示例日志:

<ip> <ip> 27 27 <ip> HTTP/1.1 - GET 8380 - GET /healthcheck/healthcheck.do HTTP/1.1 200 - [12/Nov/2019:00:33:49 +0000] - /healthcheck/healthcheck.do houston.hp.com 0 0.000 default task-245 "-" "-" "-" "-" "-" "-"
<ip> <ip> 42 42 <ip> HTTP/1.1 - POST 8743 - POST /ContactServices/api/contact/create HTTP/1.1 200 - [12/Nov/2019:07:00:54 +0000] - /ContactServices/api/contact/create - 1969 1.969 default task-199 "-" "application/json" "-" "-" "-" "-"

logstash.conf文件:

input {
 beats {
                port => 5044
                host => "<host_name>"
        }
}

filter {
  grok {
        match => ["message", '%{IPV4:remoteIP}\s+%{IPV4:localIP}\s+%{INT:throughtputData:int}\s+%{INT}\s+%{IPV4}\s+%{DATA:requestProtocol}\s+%{DATA:remoteLogicalUserName}\s+%{DATA:requestMethod}\s+%{DATA:port}\s+%{DATA}\s+%{DATA}\s+/ContactServices/api/%{DATA}\s+%{DATA:requestProtocol2}\s+%{INT:requestStatusCode}\s+%{DATA:userSessionID}\s+\[%{HTTPDATE:logTimeStamp}\]\s+%{DATA:remoteUser}\s+/ContactServices/api/%{DATA:requestedURL2}\s+%{DATA:serverName}\s+%{INT:timeTakenInMilliSec:int}\s+%{NUMBER}\s+default\s+task-%{INT}\s+"%{DATA:authorization}"\s+"%{DATA}"\s+"%{DATA}"\s+"%{DATA}"\s+"%{DATA}"\s+"%{DATA}"']
 }

        if "_grokparsefailure" in [tags]{
                drop {}
          }

        if "_groktimeout" in [tags]{
                drop {}
          }

        date {
                match => ["logTimeStamp" ,"dd/MMM/yyyy:HH:mm:ss Z" ]
        }
        mutate{
          remove_field => ["message","host","input","type","@version","prospector","beat","garbageData","offset"]
        }
}

output {
  elasticsearch {
    hosts => ["<ip>:9202"]
    index => "contact-logs-%{+YYYY.MM.dd}"
 }
}

Filebeat.conf文件

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/logs/*.log
  exclude_lines: ['.*healthcheck.*','.*swagger.*']

output.logstash:
  hosts: ["<serverip>:5044"]
  

另外,还有一个问题。

即使已创建索引,也不会解析所有有效日志

例如日志文件是否有100 correct log行(如grok filter模式的 logstash.conf文件)only 60%-70% data在Elasticsearch中显示为文档。 40% data is getting dropped周围..我不知道是什么确切原因。

如果我使用指定的grok模式检查unparsed logs中的grok debugger,则它是完美解析

对这个问题有什么解决办法吗?

0 个答案:

没有答案