如何使用rsyslog将通配符文件和文件夹中找到的文件保存到中央服务器上的正确文件名？

时间：2018-10-26 15:45:30

标签： rsyslog rainerscript

我正在尝试使用Rainerscript和rsyslog v8.38通过通配符文件夹中的通配符文件和通配符文件来捕获我的服务器日志，然后将它们保存在远端的相同文件夹结构中。感谢文章here，我可以使用通配符，但是我试图将这一概念扩展到也可以用于通配符文件夹。

目前，我可以正确地从文件夹中收集文件，但是一旦保存，所有文件中的行就会保存到每个文件夹的一个文件中，与该文件夹的名称相同，例如，如果我这样做服务器：

echo "TEST1" >> /srv/log/test-new/test.log
echo "TEST1" >> /srv/log/test-new/test-new.log

我最终在中央服务器上找到了这个

# cat /srv/rsyslog/2018/HOSTNAME/10/26/test-new

<133>2018-10-26T15:32:37.975449+00:00 HOSTNAME  test-new/test.log nested-srv-logs TEST1
<133>2018-10-26T15:32:51.042633+00:00 HOSTNAME  test-new/test-new.log nested-srv-logs TEST1

我希望我可以将文件保存在中央服务器上，该文件结构与在发送机上找到的文件夹结构相同，这可能吗？

我的配置如下，发送机具有：

module(load="omrelp")
module(load="omfwd")

template(name="CustomForwardFormat" type="list") {
    constant(value="<")
    property(name="pri")
    constant(value=">")
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    property(name="hostname")
    constant(value=" ")
    property(name=".suffix")
    constant(value=" ")
    property(name="syslogtag" position.from="1" position.to="32")
    property(name="msg" spifno1stsp="on" )
    property(name="msg")
    constant(value="\n")
    }

ruleset(name="sendToLogserver") {
    action(type="omrelp" Target="rsyslog" Port="25014" template="CustomForwardFormat"
    queue.type="LinkedList" queue.size="10000" queue.filename="q_sendToLogserver" queue.highwatermark="9000"
    queue.lowwatermark="50" queue.maxdiskspace="500m" queue.saveonshutdown="on" action.resumeRetryCount="-1"
    action.reportSuspension="on" action.reportSuspensionContinuation="on" action.resumeInterval="10")
}

ruleset(name="sendToJsonLogserver") {
    action(type="omfwd" protocol="tcp" Target="logstash" Port="5114" template="RSYSLOG_SyslogProtocol23Format"
    queue.type="LinkedList" queue.size="10000" queue.filename="q_sendToJsonLogserver" queue.highwatermark="9000"
    queue.lowwatermark="50" queue.maxdiskspace="500m" queue.saveonshutdown="on" action.resumeRetryCount="-1"
    action.reportSuspension="on" action.reportSuspensionContinuation="on" action.resumeInterval="10")
}

input(type="imfile"
    File="/srv/log/*.log"
        Tag="srv-logs"
        Ruleset="srv_logs"
        addMetadata="on")

input(type="imfile"
    File="/srv/log/*/*.log"
        Tag="nested-srv-logs"
        Ruleset="srv_logs"
        addMetadata="on")

ruleset(name="srv_logs") {
    # http://www.rsyslog.com/doc/v8-stable/rainerscript/functions.html
          # re_extract(expr, re, match, submatch, no-found)
    set $.suffix=re_extract($!metadata!filename, "(.*)/([^/]*)", 0, 2, "unknown.log");
    if ( $programname == "nested-srv-logs" ) then {
      set $.sub-suffix=re_extract($!metadata!filename, "(.*)/([^/]*)/(.*)", 0, 2, "unknown.log");
        set $.suffix=$.sub-suffix & "/" & $.suffix;
    }
    if( $!metadata!filename contains 'json' ) then {
            call sendToJsonLogserver
    } else {
      call sendToLogserver
    }
    stop
}

中央服务器具有：

module(load="imrelp")
input(type="imrelp" port="25014" ruleset="RemoteLogProcess")

module(load="builtin:omfile" FileOwner="syslog" FileGroup="syslog" dirOwner="syslog" dirGroup="syslog" FileCreateMode="0644" DirCreateMode="0755")

template(name="CustomForwardFormat" type="list") {
    constant(value="<")
    property(name="pri")
    constant(value=">")
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    property(name="hostname")
    constant(value=" ")
    property(name=".suffix")
    constant(value=" ")
    property(name="syslogtag" position.from="1" position.to="32")
    property(name="msg" spifno1stsp="on" )
    property(name="msg")
    constant(value="\n")
    }

$EscapeControlCharactersOnReceive off

template(name="FloowLogSavePath" type="list") {
    constant(value="/srv/rsyslog/")
    property(name="timegenerated" dateFormat="year")
    constant(value="/")
    property(name="hostname")
    constant(value="/")
    property(name="timegenerated" dateFormat="month")
    constant(value="/")
    property(name="timegenerated" dateFormat="day")
    constant(value="/")
    property(name="$.logpath" )
}

ruleset(name="RemoteLogProcess") {
    # For facilities local0-7 set log filename from $programname field: replace __ with /
    if ( $syslogfacility >= 16 ) then
    {
        set $.logpath = replace($programname, "__", "/");
        action(type="omfile" dynaFileCacheSize="1024" dynaFile="FloowLogSavePath" template="CustomForwardFormat"
        flushOnTXEnd="off" asyncWriting="on" flushInterval="1" ioBufferSize="64k")
    } else {
        if (($syslogfacility == 0)) then {
            set $.logpath = "kern.log";
        } else if (($syslogfacility == 1)) then {
            set $.logpath = "user";
        } else if (($syslogfacility == 2)) then {
            set $.logpath = "mail";
        } else if (($syslogfacility == 3)) then {
            set $.logpath = "daemon";
        } else if (($syslogfacility == 4) or ($syslogfacility == 10)) then {
            set $.logpath = "auth.log";
        } else if (($syslogfacility == 9) or ($syslogfacility == 15)) then {
            set $.logpath = "cron";
        } else {
            set $.logpath ="syslog";
        }
        # Built-in template RSYSLOG_FileFormat: High-precision timestamps and timezone information
        action(type="omfile" dynaFileCacheSize="1024" dynaFile="FloowLogSavePath" template="CustomForwardFormat"
        flushOnTXEnd="off" asyncWriting="on" flushInterval="1" ioBufferSize="64k")
    }
}

2 个答案:

答案 0 :(得分：1)

我看到你在这里插入了空间：

property(name=".suffix")
constant(value=" ")
property(name="syslogtag" position.from="1" position.to="32")

空格是分隔符，因此可能会使您的设置混乱。

此外，您以错误的顺序连接它们：$.suffix是文件名和目录名，我认为应该排在最后。而且您只使用syslogtag中的1-32个符号，为什么？

答案 1 :(得分：1)

检查完配置后，我以前的回答是错误的，为了使通配符文件夹正常工作，我执行了以下操作：

发件人：

input(type="imfile"
    File="/srv/log/*.log"
    Tag="srv-logs"
    Ruleset="send_sorted"
    addMetadata="on")

input(type="imfile"
    File="/srv/log/*/*.log"
    Tag="nested-srv-logs"
    Ruleset="send_sorted"
    addMetadata="on")

module(load="omrelp")

ruleset(name="send_sorted") {
    set $.suffix=substring($!metadata!filename, 9, 150);
    if( $!metadata!filename contains 'json' ) then {
      call sendToJsonLogserver
    } else {
      call sendToLogserver
    }
    stop
}

template(name="CustomForwardFormat" type="list") {
    constant(value="<")
    property(name="pri")
    constant(value=">")
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    property(name="hostname")
    constant(value=" ")
    property(name=".suffix")
    constant(value=" ")
    property(name="syslogtag")
    property(name="msg" spifno1stsp="on" )
    property(name="msg")
    constant(value="\n")
    }
ruleset(name="sendToLogserver") {
    action(type="omrelp"
           target="rsyslog"
           port="25014"
           template="CustomForwardFormat"
           queue.type="LinkedList" 
           queue.size="10000"
           queue.filename="q_sendToLogserver"
           queue.highwatermark="9000"
           queue.lowwatermark="50"
           queue.maxdiskspace="500m"
           queue.saveonshutdown="on" 
           action.resumeRetryCount="-1"
           action.reportSuspension="on"
           action.reportSuspensionContinuation="on"
           action.resumeInterval="10")
}

ruleset(name="sendToJsonLogserver") {
    action(type="omfwd" 
           target="logstash" 
           protocol="tcp" 
           port="5114" 
           template="RSYSLOG_SyslogProtocol23Format"
           queue.type="LinkedList"
           queue.size="10000"
           queue.filename="q_sendToJsonLogserver"
           queue.highwatermark="9000"
           queue.lowwatermark="50"
           queue.maxdiskspace="500m" 
           queue.saveonshutdown="on"
           action.resumeRetryCount="-1"
           action.reportSuspension="on"
           action.reportSuspensionContinuation="on"
           action.resumeInterval="10")
}

在我的中央服务器上：

module(load="imrelp")
input(type="imrelp" port="25014" ruleset="RemoteLogProcess")

module(load="builtin:omfile" FileOwner="syslog" FileGroup="syslog" dirOwner="syslog" dirGroup="syslog" FileCreateMode="0644" DirCreateMode="0755")

template(name="CustomForwardFormat" type="list") {
    constant(value="<")
    property(name="pri")
    constant(value=">")
    property(name="timestamp" dateFormat="rfc3339")
    constant(value=" ")
    property(name="hostname")
    constant(value=" ")
    property(name=".suffix")
    constant(value=" ")
    property(name="syslogtag")
    property(name="msg" spifno1stsp="on" )
    property(name="msg")
    constant(value="\n")
    }

template(name="FloowLogSavePath" type="list") {
    constant(value="/srv/rsyslog/")
    property(name="timegenerated" dateFormat="year")
    constant(value="/")
    property(name="hostname")
    constant(value="/")
    property(name="timegenerated" dateFormat="month")
    constant(value="/")
    property(name="timegenerated" dateFormat="day")
    constant(value="/")
    property(name=".logpath")
}

template(name="extract" type="string" string="%syslogtag%")

ruleset(name="RemoteLogProcess") {
    if ( $syslogfacility >= 16 ) then
    {
        set $.logpath = exec_template("extract");
        action(type="omfile" 
               dynaFileCacheSize="1024" 
               dynaFile="FloowLogSavePath" 
               template="CustomForwardFormat"
               flushOnTXEnd="off"
               asyncWriting="on"
               flushInterval="1" 
               ioBufferSize="64k")
    } else {
        if (($syslogfacility == 0)) then {
          set $.logpath = "kern.log";
        } else if (($syslogfacility == 1)) then {
            set $.logpath = "user";
        } else if (($syslogfacility == 2)) then {
            set $.logpath = "mail";
        } else if (($syslogfacility == 3)) then {
            set $.logpath = "daemon";
        } else if (($syslogfacility == 4) or ($syslogfacility == 10)) then {
            set $.logpath = "auth.log";
        } else if (($syslogfacility == 9) or ($syslogfacility == 15)) then {
            set $.logpath = "cron";
        } else {
            set $.logpath ="syslog";
        }
        action(type="omfile"
               dynaFileCacheSize="1024"
               dynaFile="FloowLogSavePath"
               template="CustomForwardFormat"
               flushOnTXEnd="off"
               asyncWriting="on"
               flushInterval="1"
               ioBufferSize="64k")
    }
}
相关问题
最新问题