Question

我试图从tomcat日志中提取错误|异常。由于tomcat日志在多行中有错误详细信息。我想将每个新的日志条目都视为我的记录，即日期可以作为记录分隔符。

Oct 4, 2012 4:00:38 PM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already.  Could not load com.google.common.base.Stopwatch.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.
java.lang.IllegalStateException
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1531)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)
        at com.myweb.ontest.stats.RpcInterceptor.intercept(RpcInterceptor.java:45)
        at com.myweb.ontest.platform.SupplySource$Iface$$EnhancerByCGLIB$$6b5e8142.finalize()
        at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
        at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
        at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
Oct 4, 2012 4:00:38 PM Org.apache.catalina.loader ....

请告诉我如何使用awk / perl脚本完成。

由于

Answer 1

我首先在记录之间添加一个空行：

awk '/^[A-Z][a-z]/ && f++{print ""}1' file

因此您可以使用空白行作为记录分隔符，在后续的awk脚本中轻松处理它：

awk '/^[A-Z][a-z]/ && f++{print ""}1' file |
awk -v RS= -F'\n' '{print "Record #" NR; for (i=1;i<=NF;i++) print "Field #" i, "[" $i "]"}'
Record #1
Field #1 [Oct 4, 2012 4:00:38 PM org.apache.catalina.loader.WebappClassLoader loadClass]
Field #2 [INFO: Illegal access: this web application instance has been stopped already.  Could not load com.google.common.base.Stopwatch.  The eventual following stack trace is caused by an error thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access, and has no functional impact.]
Field #3 [java.lang.IllegalStateException]
Field #4 [        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1531)]
Field #5 [        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)]
Field #6 [        at com.myweb.ontest.stats.RpcInterceptor.intercept(RpcInterceptor.java:45)]
Field #7 [        at com.myweb.ontest.platform.SupplySource$Iface$$EnhancerByCGLIB$$6b5e8142.finalize()]
Field #8 [        at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)]
Field #9 [        at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)]
Field #10 [        at java.lang.ref.Finalizer.access$100(Finalizer.java:14)]
Field #11 [        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)]
Record #2
Field #1 [Oct 7, 2012 4:00:38 PM org.apache.catalina.loader.WebappClassLoader loadClass]
Field #2 [INFO: just a dummy record]
Field #3 [java.lang.IllegalStateException]
Field #4 [        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1531)]
Field #5 [        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1491)]
Field #6 [        at com.myweb.ontest.stats.RpcInterceptor.intercept(RpcInterceptor.java:45)]
Field #7 [        at com.myweb.ontest.platform.SupplySource$Iface$$EnhancerByCGLIB$$6b5e8142.finalize()]
Field #8 [        at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)]
Field #9 [        at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)]
Field #10 [        at java.lang.ref.Finalizer.access$100(Finalizer.java:14)]
Field #11 [        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)]

您没有发布您希望输出的内容，如果以上内容没有给您足够的信息，请为您的示例输入发布预期输出

Answer 2

假设您要将这些错误放在MySQL数据库中，我会这样做。

#!/usr/bin/perl -Tw

use strict;
use warnings;
use English qw( -no_match_vars $OS_ERROR );
use Readonly;
use DBI;

Readonly my $LOG_FILENAME => '/var/log/tomcat.log';

Readonly my $DATE_REGEX => qr{
    \w{3} \s \d+, \s        # Oct 4,
    \d{4}         \s        # 2012
    \d+:\d+:\d+   \s \w{2}  # 4:00:38 PM
}xms;

my $sth;
{
    my $dbh = DBI->connect(
        'DBI:mysql:database=errors;host=localhost',
        'error_monitor',
        '*********',
        { 'RaiseError' => 1 }
    );

    die $DBI::errstr
        if !$dbh;

    $sth = $dbh->prepare_cached( q{
        INSERT INTO error (
            date,
            text
        ) VALUES (
            ?,
            ?
        )
    } );

    die $DBI::errstr
        if !$sth;
}

my $fh;
{
    open $fh, '<', $LOG_FILENAME
        or die "open $LOG_FILENAME: $OS_ERROR";
}

my ( $count, %record ) = ( 0, () );

while ( my $line = <$fh> ) {

    if ( $line =~ m{\A ( $DATE_REGEX ) \s ( .+ ) }xms ) {

        my ( $date, $text ) = ( $1, $2 );

        if (%record) {
            $sth->execute( @record{qw( date text )} );
            %record = ();
            $count++;
        }

        @record{qw( date text )} = ( $date, $text );
    }
    elsif ( exists $record{date} ) {

        $record{text} .= $line;
    }
    else {

        warn "malformed message: $line";
    }
}

if (%record) {
    $sth->execute( @record{qw( date text )} );
    $count++;
}

close $fh
    or die "close $LOG_FILENAME: $OS_ERROR";

print "inserted $count error records\n"
    or die "print: $OS_ERROR";

__END__

Answer 3

在perl中你可以这样做： -

use strict;
my $log = join('', <STDIN>);
my %entries = split(/([\r\n]*\w{3} \d+, \d+ \d+:\d+:\d+ [AP]M )/m, $log);

foreach my $key(keys %entries)
{
    print "$key\n";
    print "$entries{$key}\n";
}

上面的代码从stdin中读取整个日志，在日期中分割它并将日期，条目存储到散列中并最后打印散列。

使用awk / perl过滤tomcat日志

3 个答案: