Question

我是rsyslog，远程日志和弹性搜索的新手。

我配置了一个python脚本（从docker容器运行），通过TCP将日志记录发送到$ HOST：$ PORT。

我已经安装了rsyslog，模块mmnormalize和模块omelasticsearch。

现在我想了解应该如何使用我的rsyslog.conf（在主机上）来收集带有elasticsearch的日志（来自172.17.0.0/16）。

谢谢！

Answer 1

以下是我解决问题的方法：

# /etc/rsyslog.d/docker.rb
version=2
# My sample record
# [Apr 25 12:00]$CONTAINER_HOSTNAME:INFO:Package.Module.Sub-Module:Hello World
#
# Here there is the rule to parse the log records into trees
rule=:[%date:char-to:]%]%hostname:char-to::%:%level:char-to::%:%file:char-to::%:%message:rest%
#
# alternative to set date field in rfc3339 format
# rule=:[%date:date-rfc3339%]%hostname:char-to::%:%level:char-to::%:%file:char-to::%:%message:rest%

# /etc/rsyslog.conf
module(load="mmnormalize")
module(load="omelasticsearch")
module(load="imtcp")

# apply to log records coming from address:port the specified rule
input(type="imtcp"
      address="127.0.0.1" # $HOST
      port="514"          # $PORT
      ruleset="docker-rule")

# define the rule in two actions; parsing the log record into a tree with
# root $! ($!son-node!grandson-node...) and adding to the elasticsearch index
# 'docker-logs' the parsed tree, but in a JSON format (specified in a template)
ruleset(name="docker-rule"){
    action(type="mmnormalize"
           rulebase="/etc/rsyslog.d/docker.rb"
           useRawMsg="on"
           path="$!")
    action(type="omelasticsearch"
           template="docker-template"
           searchIndex="docker-logs"
           bulkmode="on"
           action.resumeretrycount="-1")
}

# define the template:
# 'constants' are simply putting into the record JSON delimiters as '{' or ','
# 'properties' are simply putting the values of the parsed tree into fields
# named in the previous constant statements through 'value="..."'
# the result is a JSON record like:
# { "@timestamp":"foo",
#   "hostname":"bar",
#   "level":"foo",
#   "file":"bar",
#   "message":"foo"
# }
template(name="docker-template" type="list"){
    constant(value="{")
        constant(value="\"@timestamp\":")
            constant(value="\"")
                # because kibana would use '$!date' as string not as date
                # that is the only field not from the parsed tree
                property(name="timereported" dateFormat="rfc3339")
            constant(value="\"")
        constant(value=",")
        constant(value="\"hostname\":")
            constant(value="\"")
                property(name="$!hostname")
            constant(value="\"")
        constant(value=",")
        constant(value="\"level\":")
            constant(value="\"")
                property(name="$!level")
            constant(value="\"")
        constant(value=",")
        constant(value="\"file\":")
            constant(value="\"")
                property(name="$!file")
            constant(value="\"")
        constant(value=",")
        constant(value="\"message\":")
            constant(value="\"")
                property(name="$!message")
            constant(value="\"")
    constant(value="}")
}

接下来安装kibana可以“配置索引模式”，只需将“索引名称或模式”设置为“docker-logs”，将“Time-field name”设置为“@timestamp”

请注意，日志源无法控制（172.17.0.0/16）;每个日志记录发送到$ HOST：$ PORT如果正确解析将被插入到elasticsearch索引中。

配置Rsyslog（Docker-＆gt; TCP-＆gt; Rsyslog-＆gt; ElasticSearch）

1 个答案: