Question

我有一个调试日志文件，如下所示：

示例文件：

DEBUG: Fri Dec  7 06:49:14 2018:16920 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16920: start <ID>
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "output
output output
output"
DEBUG: extra lines

我只想获取ID和最终输出，如下所示。

预期输出：

<ID> "output
output output
output"

我想用python或bash做到这一点。任何帮助，将不胜感激。谢谢

当前代码仅适用于“最终输出”。但是我也想获取ID，并且应该有一种方法可以区分（分隔符）每个ID及其输出。

stream=open("debuglog.txt","r")
lines=stream.readlines()

flag = 0
for i in lines:
    if "DEBUG:" in i:
        flag = 0
    if "final output is" in i:
        flag = 1
    if flag:
        print(i)

Answer 1

示例日志文件：

DEBUG: Fri Dec  7 06:49:14 2018:16920 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16920: start 12324
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "output output output output"
DEBUG: extra lines

请找到代码。另外，我假设每个ID和输出只有一个实例

import sys, re

stream=open("log","r")
lines=stream.readlines()


flag_ID = 0
flag_output = 0
flag_print = 1
for i in lines:
    ID = re.match("DEBUG: [\w :]* start (\d+)", i)
    output = re.match("DEBUG: [\w :]* Final output is \"([\w ]*)\"", i)
    if ID:
        flag_ID = 1
        value_ID = ID.group(1)
    if output:
        flag_output = 1 
        value_output = output.group(1)
    if flag_output == 1 and flag_ID == 1 and flag_print == 1:
        print "{0} {1}".format(value_ID, value_output)
        flag_print = 0

输出

12324 output output output output

请打勾并接受是否可以解决您的问题;）

Answer 2

使用python，怎么样：

#!/usr/bin/python

import re
text = open("logfile", "r").read()

regex = r'start (.+?)$.*?Final output is (.+?)(?:(?=\nDEBUG)|\Z)'
for m in re.finditer(regex, text, re.MULTILINE|re.DOTALL):
    for i in m.groups():
            print(i.replace('\n', ' '))

输入日志文件：

DEBUG: Fri Dec  7 06:49:14 2018:16920 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16920: start <ID>
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "output
output output
output"
DEBUG: extra lines

DEBUG: Fri Dec  7 06:49:14 2018:16920 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16920: start <ID2>
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "output2
output+ output/
output2"

输出：

<ID>
"output output output output"
<ID2>
"output2 output+ output/ output2"

正则表达式中的第一个Paren捕获start之后和换行符之前的所有字符，并将字符串存储到1st group中。
正则表达式中的第二个括号也捕获Final output is之后，DEBUG之前或字符串末尾的所有字符，并将字符串存储到2nd group。由于re.DOTALL选项，字符串中可以包含换行符。
第三个parren为空长度锚点，不包含在捕获组中。

编辑

下面的更新版本可为单个ID处理多个“最终输出”，仅显示每个ID的最后一个输出：

#!/usr/bin/python

import re
text = open("logfile", "r").read()

regex = r'start (.+?)$(.+?)(?:(?=DEBUG[^\n]+?start)|\Z)+'
regex2 = r'Final output is (.+?)(?:(?=\nDEBUG)|\Z)'

for m in re.finditer(regex, text, re.MULTILINE|re.DOTALL):
    print m.group(1)
    m2 = re.finditer(regex2, m.group(2), re.MULTILINE|re.DOTALL)
    print list(m2).pop().group(1).replace('\n', ' ')

输入日志文件：

DEBUG: Fri Dec  7 06:49:14 2018:16920 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16920: start <ID1>
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "output
output output
output"
DEBUG: extra lines
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "this
is the last output
for <ID1>"
DEBUG: extra lines

DEBUG: Fri Dec  7 06:49:14 2018:16920 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16920: start <ID2>
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "output2
output+ output/
output2"

并输出：

<ID1>
"this is the last output  for <ID1>"
<ID2>
"output2 output+ output/ output2"

我将子字符串的提取分为两个步骤：

提取ID和剩余文本（可能包含额外的字符串）。这是通过regex处理的。
从上面的“剩余文本”中提取“最终输出”子字符串。这是通过regex2处理的。

然后选择最后一个“最终输出”并显示。

编辑

以下版本禁止显示包含某些关键字的消息：

#!/usr/bin/python

import re
text = open("logfile", "r").read()

exclude = 'xyz'     # keyword to suppress the output

regex = r'start (.+?)$(.+?)(?:(?=DEBUG[^\n]+?start)|\Z)+'
regex2 = r'Final output is (.+?)(?:(?=\nDEBUG)|\Z)'
#regex = r'start (.+?)$.*?Final output is (.+?)(?=\nDEBUG)'
#for m in re.finditer(regex, text, flags=(re.MULTILINE|re.DOTALL)):
for m in re.finditer(regex, text, re.MULTILINE|re.DOTALL):
    print m.group(1)
    m2 = re.finditer(regex2, m.group(2), re.MULTILINE|re.DOTALL)
    message = list(m2).pop().group(1).replace('\n', ' ')
    if message.count(exclude):
        print 'error:' + exclude
    else:
        print message

示例日志文件：

DEBUG: Fri Dec  7 06:49:14 2018:16920 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16920: start <ID1>
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "output
output output
output"
DEBUG: extra lines
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "this
is the last output
for ID1"
DEBUG: extra lines

DEBUG: Fri Dec  7 06:49:14 2018:16920 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16920: start <ID2>
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "output2
output+ output/
output2"
DEBUG: extra lines

DEBUG: Fri Dec  7 06:49:14 2018:16920 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16920: start <ID3>
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "this message
contains the word xyz"
DEBUG: extra lines

输出：

<ID1>
"this is the last output  for ID1"
<ID2>
"output2 output+ output/ output2"
<ID3>
error:xyz

Answer 3

如果文件可以放入内存，则使用Perl，您可以使用单行格式。

/tmp> cat debug.log
DEBUG: Fri Dec  7 06:49:14 2018:16920 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16920: start <ID1>
DEBUG: Fri Dec  7 06:49:14 2018:16920: Final output is "output
output output
output"
DEBUG: extra lines
DEBUG: Fri Dec  7 06:49:14 2018:16921 extra text
DEBUG: Fri Dec  7 06:49:14 2018:16921: start <ID2>
DEBUG: Fri Dec  7 06:49:14 2018:16921: Final output is "output output output output"
DEBUG: extra lines
/tmpl>
/tmp> perl -0777 -ne ' while(/^DEBUG(.+?)start (\S+).*?DEBUG.+?Final output is \"(.+?)\"/smg) { print "$2 $3\n" } ' debug.log
<ID1> output
output output
output
<ID2> output output output output
/tmp>

通过应用条件解析日志文件

3 个答案: