Question

We are attempting to capture logs to syslog-ng over tcp connection. The logs looks something like this:

1810717353--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00--<11>Oct 23 16:03:23 HOSTNAME [443-Multiplexer][0x80000001][xsltmsg][error] xmlfirewall(443-Multiplexer): trans(2607607975)[error][ip] gtid(value):
1810717354--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00--                    Default rule caught error code '
1810717355--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00--                    0x00230001
1810717356--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00--                    '
1810717357--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00--
1810717358--user--notice--IPV4----2017-10-23T16:03:23.015170-04:00--<14>Oct 23 16:03:23 HOSTNAME [443-Multiplexer][value][multistep][info] xmlfirewall(443-Multiplexer): trans(2607607975)[error][ip] gtid(2607607975):

Is there any way to strip out the syslog header during the input? If the logs remain multiline, thats fine, we can package them with logstash during read, but we need a way for the logs to now have a header when they are written to file ..

Here is my current syslog-ng config :

@version: 3.6
#@include "scl.conf"

options {
        flush_lines (0);
        frac-digits(8);
        use-rcptid (yes);
        time_reopen (10);
        log_fifo_size (1000);
        chain_hostnames (off);
        use_dns (no);
        use_fqdn (no);
        create_dirs (no);
        keep_hostname (yes);
        use-rcptid (yes);
        threaded(yes);
        log_msg_size(32768);
};
    source sourcename
    {
       network(ip("0.0.0.0") port(5521) flags(no-parse)); 
    };
    destination destname
    {
        file("/opt/elk/data/syslogs/datapower/${HOST}.log"
            #follow-freq(0) multi-line-mode(regexp) multi-line-prefix("^\<\[0-9]{2}\>[A-z]{3} [0-9]{2} [0-9]{2}\:[0-9]{2}\:[0-9]{2}")
            create_dirs(yes) dir_owner("elkuser") dir_group("users") dir_perm(0700)
            owner("user") group("group") perm(0600)
            template("${RCPTID}--$FACILITY--$PRIORITY--$FULLHOST--$PROGRAM--$ISODATE--${MSGHDR}${MSG}\n") 
        );
    };

Thanks

Answer 1

我不确定您要拆分邮件的位置，但根据具体情况，您可以执行以下操作：

如果足以删除固定数量的字符，则可以使用substr模板函数（https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-template-functions.html）
如果您想加入多行消息，可能需要检查网络来源的多行选项：https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-source-network.html
如果您需要对这些日志进行大量自定义处理，那么编写自定义python解析器（https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/python-parser.html）可能会更容易。请注意，Python解析器需要最新版本的syslog-ng。

此外，您似乎以某种方式将日志发送到Elasticsearch（至少我怀疑目的地中的elk代表的是什么）：最近版本的syslog-ng可以直接执行此操作：{{3} }

HTH，罗伯特

Syslog-ng multiline input over TCP/Network module

1 个答案: