我正在尝试使用LogMine自动生成GROK模式
日志示例:
Error IGXL error [Slot 2, Chan 16, Site 0] HSDMPI:0217 : TSC3 Fifo Edge EG0-7 Underflow. Please check the timing programming. Edge events should be fired in the sequence and the time between two edges should be more than 2 MOSC ticks.
Error IGXL error [Slot 2, Chan 18, Site 0] HSDMPI:0217 : TSC3 Fifo Edge EG0-7 Underflow. Please check the timing programming. Edge events should be fired in the sequence and the time between two edges should be more than 2 MOSC ticks.
对于上述日志,我得到以下模式:
re.compile('^(?P<Event>.*?)\\s+(?P<Tester>.*?)\\s+(?P<State>.*?)\\s+(?P<Slot>.*?)\\s+(?P<Instrument>.*?)\\s+(?P<Content1>.*?):\\s+(?P<Content>.*?)$')
但是我希望看到这样的Grok Pattern(Logstash):
%{LOGLEVEL:level} *%{DATA:Instrument} %{LOGLEVEL:State} \[%{DATA:slot} %{DATA:slot} %{DATA:channel} %{DATA:channel} %{DATA:Site}] %{DATA:Tester} : %{DATA:Content}
代码:LogMine是从以下链接导入的:https://github.com/logpai/logparser/tree/master/logparser/LogMine
import sys
import os
sys.path.append('../')
import LogMine
input_dir ='E:\LogMine\LogMine' # The input directory of log file
output_dir ='E:\LogMine\LogMine/output/' # The output directory of parsing results
log_file ='E:\LogMine\LogMine/log_teradyne.txt' # The input log file name
log_format ='<Event> <Tester> <State> <Slot> <Instrument><content> <contents> <context> <desc> <junk> ' # HDFS log format
levels =1 # The levels of hierarchy of patterns
max_dist =0.001 # The maximum distance between any log message in a cluster and the cluster representative
k =1 # The message distance weight (default: 1)
regex =[] # Regular expression list for optional preprocessing (default: [])
print(os.getcwd())
parser = LogMine.LogParser(input_dir, output_dir, log_format, rex=regex, levels=levels, max_dist=max_dist, k=k)
parser.parse(log_file)
此代码仅返回已解析的CSV文件,我希望生成GROK模式,并稍后在Logstash应用程序中使用它来解析日志。