继续Groking On Failure

时间：2016-03-09 22:35:26

标签： elasticsearch logstash kibana logstash-grok

我有一个grok表达式切片我的log4j文件，通过弹性搜索使其可用于kibana。我从一个简单的grok表达开始，因为我还在学习

match => {"message" => "%{TIMESTAMP_ISO8601:timestamp}\s+%{LOGLEVEL:loglevel}\s+%{IP:ip}"}

在这种情况下，我的一些日志文件没有IP文件。所以grok调试器显示“不匹配”。

这是否意味着我会错过被解析的特定行会因为这些匹配而丢失？

即使存在一些不匹配，如何继续匹配过程？

logstash

创建的示例日志消息 
 Eg1. Without IP   {"@timestamp":"2016-03-09T22:54:13.103Z","message":"2013-04-05 00:00:02,101 ERROR [scheduler_Worker-6          ]                 (DataProcessor.java:412 ) RemoteException > \nAxisFault\n faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server\n faultSubcode: \n faultString: 0005: No Data matched the criteria Specified\n faultActor: \n faultNode: \n faultDetail: \n\t{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>GetTripsByFlightNumber</con:node><con:pipeline>GetTripsByFlightNumber_response</con:pipeline><con:stage>Create Get Trips By Flight Number Response</con:stage><con:path>response-pipeline</con:path></con:location>\n0005: No Data matched the criteria Specified\n\tat org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)\n\tat org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)\n\tat org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat javax.xml.parsers.SAXParser.parse(Unknown Source)\n\tat org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)\n\tat org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)\n\tat org.apache.axis.Message.getSOAPEnvelope(Message.java:435)\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)\n\tat org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)\n\tat org.apache.axis.client.Call.invokeEngine(Call.java:2784)\n\tat org.apache.axis.client.Call.invoke(Call.java:2767)\n\tat org.apache.axis.client.Call.invoke(Call.java:2443)\n\tat org.apache.axis.client.Call.invoke(Call.java:2366)\n\tat org.apache.axis.client.Call.invoke(Call.java:1812)\n\tat com.acme.axsbagtracing.flight.ws.qantas.FlightScheduleRequestBindingStub.getTripsByFlightNumber(FlightScheduleRequestBindingStub.java:1563)\n\tat {"@timestamp":"2016-03-09T22:54:13.103Z","message":"2013-04-05 00:00:02,319 ERROR [scheduler_Worker-6          ]                 (DataProcessor.java:412 ) RemoteException > \nAxisFault\n faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server\n faultSubcode: \n faultString: 0005: No Data matched the criteria Specified\n faultActor: \n faultNode: \n faultDetail: \n\t{http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>GetTripsByFlightNumber</con:node><con:pipeline>GetTripsByFlightNumber_response</con:pipeline><con:stage>Create Get Trips By Flight Number Response</con:stage><con:path>response-pipeline</con:path></con:location>\n0005: No Data matched the criteria Specified\n\tat org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)\n\tat org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)\n\tat org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source)\n\tat com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)\n\tat javax.xml.parsers.SAXParser.parse(Unknown Source)\n\tat org.apache.axis.encoding.DeserializationContext.parse(DeserializationContext.java:227)\n\tat org.apache.axis.SOAPPart.getAsSOAPEnvelope(SOAPPart.java:696)\n\tat org.apache.axis.Message.getSOAPEnvelope(Message.java:435)\n\tat org.apache.axis.handlers.soap.MustUnderstandChecker.invoke(MustUnderstandChecker.java:62)\n\tat org.apache.axis.client.AxisClient.invoke(AxisClient.java:206)\n\tat org.apache.axis.client.Call.invokeEngine(Call.java:2784)\n\tat org.apache.axis.client.Call.invoke(Call.java:2767)\n\tat org.apache.axis.client.Call.invoke(Call.java:2443)\n\tat org.apache.axis.client.Call.invoke(Call.java:2366)\n\tat org.apache.axis.client.Call.invoke(Call.java:1812)\n\tat com.acme.axsbagtracing.flight.ws.qantas.FlightScheduleRequestBindingStub.getTripsByFlightNumber(FlightScheduleRequestBindingStub.java:1563)\n\tat com.acme.bagassist.common.scheduler.InboundListFlightDataProcessor.callOGSForFlightTime(DataProcessor.java:398)\n\tat com.acme.bagassist.common.scheduler.InboundListFlightDataProcessor.processInboundFlightData(DataProcessor.java:290)\n\tat sun.reflect.GeneratedMethodAccessor601.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)\n\tat java.lang.reflect.Method.invoke(Unknown Source)\n\tat org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:273)\n\tat org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:264)\n\tat org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)\n\tat org.quartz.core.JobRunShell.run(JobRunShell.java:203)\n\tat org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)","@version":"1","tags":["multiline","beats_input_codec_multiline_applied","_grokparsefailure"],"beat":{"hostname":"LVRJ8YRJX1","name":"LVRJ8YRJX1"},"count":1,"fields":null,"input_type":"log","offset":7568,"source":"C:\\logs\\applog_x16.log","type":"log","host":"LVRJ8YRJX1"}


 Eg2. With IP      {"@timestamp":"2016-03-09T22:54:13.103Z","message":"2013-04-05 00:07:36,535 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:504 ) WTSDK- Command: V.1\nVHDG.WA/I5BAGXS/E�/PQF7436\nVGZ.\nVQF////33080\nWM DAH PERQF11417.FAPAX/BAG/RTI/CLM/OSI","@version":"1","tags":["multiline","beats_input_codec_multiline_applied","_grokparsefailure"],"beat":{"hostname":"LVRJ8YRJX1","name":"LVRJ8YRJX1"},"count":1,"fields":null,"input_type":"log","offset":7834,"source":"C:\\logs\\applog_x16.log","type":"log","host":"LVRJ8YRJX1"}

请从我的日志文件中找到以下内容：

2013-04-05 00:00:02,101 ERROR [scheduler_Worker-6          ]                 (DataProcessor.java:412 ) RemoteException > 
AxisFault
 faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server
 faultSubcode: 
 faultString: 0005: No Data matched the criteria Specified
 faultActor: 
 faultNode: 
 faultDetail: 
    {http://www.bea.com/wli/sb/context}fault:<con:errorCode>0005</con:errorCode><con:reason>No Data matched the criteria Specified</con:reason><con:location><con:node>getNumber</con:node><con:pipeline>getNumber_response</con:pipeline><con:stage>Create Number Response</con:stage><con:path>response-pipeline</con:path></con:location>

0005: No Data matched the criteria Specified1
    at org.apache.axis.message.SOAPFaultBuilder.createFault(SOAPFaultBuilder.java:222)
    at org.apache.axis.message.SOAPFaultBuilder.endElement(SOAPFaultBuilder.java:129)
    at org.apache.axis.encoding.DeserializationContext.endElement(DeserializationContext.java:1087)
    at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(Unknown Source)
    at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(Unknown Source)
2013-04-05 00:07:36,535 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:504 ) WTSDK- Command: V.1
ACDG.WA/ACMEXS/E…/PQF7436
VQZ.
VMF////33080
WM DAH 11417.FAX/BG/RTI/CAM/OZI
2013-04-05 00:07:36,557 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:505 ) WTSDK- PID: PQF7436
2013-04-05 00:07:40,120 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:517 ) WTSDK: Response Time before parsing using PID PQF7436 == 3560 ms
2013-04-05 00:07:40,126 INFO  [TP-Processor8               ] 10.136.59.190   (                        WTSDK.java:547 ) WTSDK: Response string after parsing: WM DAH PERQF11417

谢谢， 圣

1 个答案:

答案 0 :(得分：1)

grokparsefailure意味着您提供给grok的所有模式都不能成功应用于收到的消息。其余的过滤器和输出将会运行，但是您希望由grok {}创建的任何字段都不会发生。

在正则表达式中，您可以将它们的部分视为可选，例如

(?:%{URIPATHPARAM})?

编辑：

除了可选的IP字段外，您的数据中还有相当随机的空格。 ％{SPACE}将匹配任何空格，并且通常比“\ s *”更具可读性。

调试器中的几分钟导致了这个：

%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}%{SPACE}\[%{NOTSPACE:program}%{SPACE}]%{SPACE}(?:%{IP:ip})?%{SPACE}\(%{SPACE}%{NOTSPACE:coderef} \)

适用于两个输入。

相关问题
最新问题