我正在尝试使dockerized 流利通过tcp接收一些日志行。 可以在每个日志行格式的 fluent.conf 中包含一个正则表达式 parser 块,以将其转换为流畅的 time-tag-jsonmsg 格式。
但是,我无法访问日志行本身的创建方式。他们可能看起来像这样:
10:26:30 WARNING [PyClass.method:135]: Some text.
这是我已经尝试过的:
Dockerfile没什么特别的:
FROM fluent/fluentd:v1.11-1
# This line is important. Apparently the default user is "fluent". However, root is needed for apk.
# https://github.com/fluent/fluentd-docker-image/issues/21
USER root
RUN apk add --no-cache --update --virtual .build-deps \
sudo build-base ruby-dev \
&& sudo gem install fluent-plugin-elasticsearch \
&& sudo gem sources --clear-all \
&& apk del .build-deps \
&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem
COPY log_fluentd/config/fluent.conf /fluentd/etc/
# COPY log_fluentd/config/entrypoint.sh /bin/ # << Led to trouble but there already is such a file.
# Default port for Forward
EXPOSE 24224
# Default port for TCP
EXPOSE 5170
USER fluent
docker-compose up
-通过
server_fluentd:
image: paws_server_fluentd:latest
container_name: server_fluentd
environment:
- "FLUENTD_CONF=fluent.conf"
ports:
- "24224:24224"
- "24224:24224/udp"
- "5170:5170"
networks:
- paws
volumes:
- "paws_log:/fluentd/log"
最后,这是我根据从文档派生的内容炮制的配置文件的相关部分:
<system>
workers 1
@log_level info
root_dir /fluentd/log
</system>
<source>
@type tcp
@label @mainstream
@id pawc
tag myapp.tcp # required
port 5170 # defaults to 5170
bind 0.0.0.0
# https://docs.fluentd.org/parser/regexp
<parse>
@type regexp
# Example from PAWC log line: "10:26:29 INFO [trifles.config:114]: Some text."
expression /^(?<logtime>[^\s]+) (?<loglvl>[^\s]+) \[(?<file>[^\]:]+):(?<line>\d+)\]: (?<msg>.*)$/
time_key logtime
time_format %H:%M:%S
types line:integer
</parse>
</source>
<filter **>
@type stdout
</filter>
<label @mainstream>
<match myapp.tcp>
@type file
@id output_tcp
path /fluentd/log/tcp.*.log
symlink_path /fluentd/log/tcp.log
</match>
</label>
此(和隐藏的http内容)导致此容器日志:
[info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf"
[info]: gem 'fluent-plugin-elasticsearch' version '4.1.1'
[info]: gem 'fluentd' version '1.11.1'
[warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
[info]: using configuration file: <ROOT> [.. shortened ..]
[info]: starting fluentd-1.11.1 pid=6 ruby="2.5.8"
[info]: spawn command to main: cmdline=["/usr/bin/ruby", "-Eascii-8bit:ascii-8bit", "/usr/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--under-supervisor"]
[info]: adding match in @mainstream pattern="docker.**" type="file"
[info]: adding match in @mainstream pattern="myapp.tcp" type="file"
[info]: adding match in @mainstream pattern="myapp.access" type="file"
[info]: adding filter pattern="**" type="stdout"
[info]: adding source type="tcp"
[info]: adding source type="http"
[warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
[info]: #0 starting fluentd worker pid=20 ppid=6 worker=0
[info]: #0 fluentd worker is now running worker=0
fluent.info: {"pid":20,"ppid":6,"worker":0,"message":"starting fluentd worker pid=20 ppid=6 worker=0"}
[warn]: #0 no patterns matched tag="fluent.info"
fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0"}
[warn]: #0 no patterns matched tag="fluent.info"
然后在我的主机上启动python 3.7说
data_to_send = bytes("10:26:30 WARNING [ServiceBase.subservice:135]: Some text.", 'utf-8')
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost', 5170))
s.sendall(data_to_send)
s.close()
使我毫无反应。容器中至少有一个(空)文件树:
/fluentd/log # tree
.
├── http.log -> /fluentd/log/worker0/output_http/buffer/buffer.b5ac49abfabe305660f691eb4d5e782c2.log
└── worker0
[.. shortened ..]
└── output_tcp
└── buffer
13 directories, 3 files [none of them about tcp]
我怀疑这与标记有关。你看到我想念的吗?
答案 0 :(得分:0)
实际上,一切都很简单:regexp希望将行尾\n
与它的$
字符进行匹配。