Question

对于logstash解析多行日志消息我还是一个新手仍然看起来有点令人生畏但是我希望我正在尝试做的事情（也就是解析logstash日志）是经过试验和测试的模式，希望有人能够指出我的工作解决方案。

基本上我有这样的错误：

[2014-10-15 22:34:36,958][DEBUG][action.search.type       ] [kafka] [logstash-2014.10.15][1], node[RBRz8xNgQQKsAnEgbjqVTw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4511ca8d] lastShard [true]
org.elasticsearch.search.SearchParseException: [logstash-2014.10.15][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"facets":{"stats":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:200"}},{"query_string":{"query":"status:304"}},{"query_string":{"query":"_missing_:status"}},{"query_string":{"query":"type:\"nginx-error\""}},{"query_string":{"query":"type:(-\"nginx-*\")"}},{"query_string":{"query":"status:[308 TO 999]"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577151,"to":1413412477151}}}]}}}}}}},"stats_Success":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:200"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577151,"to":1413412477151}}}]}}}}}}},"stats_Cache Hits":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:304"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577151,"to":1413412477151}}}]}}}}}}},"stats_No Status":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"_missing_:status"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}},"stats_NGINX error":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"type:\"nginx-error\""}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}},"stats_Not Nginx":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"type:(-\"nginx-*\")"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}},"stats_Failure":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:[308 TO 999]"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}}},"size":0}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:660)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:516)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:488)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet [stats]: field [host] isn't a number field, but a string
    at org.elasticsearch.search.facet.statistical.StatisticalFacetParser.parse(StatisticalFacetParser.java:132)
    at org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:644)
    ... 9 more

我似乎在使用＆＃34; host＆＃34;在Kibana仪表板中作为数字。我实际上并不是对实际错误感兴趣并且更有兴趣能够以有效的logstash方式解析它，以便我可以查看Kibana并查看合理的错误消息。

我假设它会涉及一些multiline { ... }编解码器用于我的文件输入，但看着那条消息看起来很毛茸茸......肯定有一个谚语＆＃34;轮＆＃ 34;在那里，我不需要发明什么？

Answer 1

multiline {...}是要走的路。请尝试file input类型中的以下代码段。

codec => multiline
{
    pattern => "(^\d+\serror)|(^.+Exception: .+)|(^\s+at .+)|(^\?s+... \d+ more)|(^\s*Caused by:.+)|(^.+ \d+ common frames omitted)
    what => "previous"
    multiline_tag => "multi_tagged"
}

Answer 2

每个例外都以日期开始，因此关键。这是一个＆＃39;过滤器＆＃39;例如，但它适用于编解码器：

filter {
    multiline {
      negate => 'true'
      pattern => "^\[%{YEAR}"
      what => 'previous'
    }
  }
}

Grok多线配方？

2 个答案: