我正在使用图像gcr.io/google-containers/fluentd-elasticsearch(v2.3.1),以便使fluentd收集一些日志并将其发送到Elastic search。我将以下配置用于fluentd:
<source>
type forward
port {{.Values.fluentd.forward.port}}
bind 0.0.0.0
</source>
<filter kube.**>
@type parser
@log_level debug
key_name log
reserve_data true
remove_key_name_field true
<parse>
@type json
time_key time
time_type string
time_format %iso8601
</parse>
</filter>
<filter kube.**>
@type record_transformer
@log_level debug
enable_ruby
<record>
kubernetes ${record["kubernetes"]["cluster_name"] = "{{.Values.clusterName}}"; record["kubernetes"] }
logtrail {"host": "${record['kubernetes']['pod_name']}", "program":"${record['kubernetes']['container_name']}"}
</record>
</filter>
<filter kube.**>
@type concat
key log
stream_identity_key kubernetes["docker_id"]
multiline_end_regexp /\n$/
separator ""
</filter>
上面列出的配置应该解析与名为log的键关联的JSON。但是我看到JSON根本没有解析。以下是我流畅进行过滤后得到的JSON。我曾预计与键日志关联的JSON将被解析。
{"kubernetes":{"pod_name":"api-dummy-dummy-vcpqr","namespace_name":"dummy","pod_id":"dummy","labels":{"name":"api-dummy","pod-template-hash":"dummy","tier":"dummy"},"host":"dummy","container_name":"api-dummy","docker_id":"dummy","cluster_name":"dummy Dev"},"log":"{\"name\":\"dummy\",\"json\":false,\"hostname\":\"api-dummy-dummy-vcpqr\",\"pid\":24,\"component\":\"dummy\",\"level\":30,\"version\":\"1.0\",\"timestamp\":1539645856126}","stream":"stdout","logtrail":{"host":"api-dummy-dummy-vcpqr","program":"api-dummy"}}
我花了3天多的时间为此找到解决方案。我什至尝试使用https://github.com/edsiper/fluent-plugin-docker,但这没有帮助。尽管该插件有助于解析JSON,但它导致解析的日志消息被我的Elastic搜索拒绝。
答案 0 :(得分:0)
您的日志字段不是有效的JSON。
{
"kubernetes": {
"pod_name": "api-dummy-dummy-vcpqr",
"namespace_name": "dummy",
"pod_id": "dummy",
"labels": {
"name": "api-dummy",
"pod-template-hash": "dummy",
"tier": "dummy"
},
"host": "dummy",
"container_name": "api-dummy",
"docker_id": "dummy",
"cluster_name": "dummy Dev"
},
"log": "{\"name\":\"dummy\",\"json\":false,\"hostname\":\"api-dummy-dummy-vcpqr\",\"pid\":24,\"component\":\"dummy\",\"level\":30,\"version\":\"1.0\",\"timestamp\":1539645856126",
"stream": "stdout",
"logtrail": {
"host": "api-dummy-dummy-vcpqr",
"program": "api-dummy"
}
}
在解析为JSON之前,您应该连接日志字段。