如何使用LogMine自动生成Grok模式

时间:2019-01-24 11:57:47

标签: logstash logstash-grok

我正在尝试使用LogMine自动生成GROK模式

日志示例:

Error   IGXL    error [Slot 2, Chan 16, Site 0] HSDMPI:0217 : TSC3 Fifo Edge EG0-7 Underflow.  Please check the timing programming.  Edge events should be fired in the sequence and the time between two edges should be more than 2 MOSC ticks.    
Error   IGXL    error [Slot 2, Chan 18, Site 0] HSDMPI:0217 : TSC3 Fifo Edge EG0-7 Underflow.  Please check the timing programming.  Edge events should be fired in the sequence and the time between two edges should be more than 2 MOSC ticks.    

对于上述日志,我得到以下模式:

re.compile('^(?P<Event>.*?)\\s+(?P<Tester>.*?)\\s+(?P<State>.*?)\\s+(?P<Slot>.*?)\\s+(?P<Instrument>.*?)\\s+(?P<Content1>.*?):\\s+(?P<Content>.*?)$') 

但是我希望看到这样的Grok Pattern(Logstash):

%{LOGLEVEL:level}    *%{DATA:Instrument} %{LOGLEVEL:State} \[%{DATA:slot} %{DATA:slot} %{DATA:channel} %{DATA:channel} %{DATA:Site}] %{DATA:Tester} : %{DATA:Content}    

代码:LogMine是从以下链接导入的:https://github.com/logpai/logparser/tree/master/logparser/LogMine

import sys    
import os    
sys.path.append('../')    
import LogMine    

input_dir  ='E:\LogMine\LogMine' # The input directory of log file    
output_dir ='E:\LogMine\LogMine/output/' # The output directory of parsing  results    
log_file   ='E:\LogMine\LogMine/log_teradyne.txt' # The input log file name    
log_format ='<Event> <Tester> <State> <Slot> <Instrument><content> <contents> <context> <desc> <junk> ' # HDFS log format     
levels     =1 # The levels of hierarchy of patterns     
max_dist   =0.001 # The maximum distance between any log message in a cluster and the cluster representative     
k          =1 # The message distance weight (default: 1)     
regex      =[]  # Regular expression list for optional preprocessing (default: [])     

print(os.getcwd())    
parser = LogMine.LogParser(input_dir, output_dir, log_format, rex=regex, levels=levels, max_dist=max_dist, k=k)     
parser.parse(log_file)    

此代码仅返回已解析的CSV文件,我希望生成GROK模式,并稍后在Logstash应用程序中使用它来解析日志。

0 个答案:

没有答案