Question

非常感谢您的帮助。

我熟练地在kubernetes上运行。我有来自Java应用程序的日志。堆栈跟踪是多行的，问题在于多行日志的每一行都被报告为新的日志事件。

下面是我的配置。有人可以建议配置有什么问题吗？另外，我正在使用容器而不是docker来运行容器。

input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        Multiline         On
        Parser_Firstline  multiline
        Parser_1          cri

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://k3s-master.test.com:6443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
    [FILTER]
        Name      parser
        Match     kube.*
        Key_Name  log
        Parser    cri
        Reserve_Data On
    [FILTER]
        Name      parser
        Match     kube.*
        Key_Name  message
        Parser    json
        Reserve_Data On
        Preserve_Key On

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${FLUENT_ELASTICSEARCH_HOST}
        Port            ${FLUENT_ELASTICSEARCH_PORT}
        Logstash_Format On
        Logstash_Prefix logstash-k3s-test
        Replace_Dots    On
        Retry_Limit     False
        Type            flb_type



parsers.conf: |
    [PARSER]
        Name   json
        Format json

[PARSER]
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On

[PARSER]
    Name        multiline
    Format      regex
    Regex       ^(?<time>\d{4}[-\/]\d{2}[-\/]\d{2}.*) (?<stream>stdout|stderr) . (?<message>(\{|(ts=)?\[?\d{4}[-\/]\d{2}[-\/]\d{2}).*)$

示例日志：

2020-10-15T08:48:37.932000736+02:00 stdout F 2020-10-15 08:48:37.923 ERROR [Service,a1793c4339d3f230,a1793c4339d3f230,true] 1 --- [nio-8080-exec-2] c.k.m.i.rest.GlobalExceptionHandler      : Status Response Error: 304 NOT_MODIFIED
2020-10-15T08:48:37.932050633+02:00 stdout F
2020-10-15T08:48:37.932061926+02:00 stdout F org.springframework.web.server.ResponseStatusException: 304 NOT_MODIFIED
2020-10-15T08:48:37.932071245+02:00 stdout F    at com.kobo.microservice.rest.InventoryController.queryInventory(InventoryController.java:70) ~[classes!/:1.197.1]
2020-10-15T08:48:37.932080389+02:00 stdout F    at jdk.internal.reflect.GeneratedMethodAccessor280.invoke(Unknown Source) ~[na:na]
2020-10-15T08:48:37.932088907+02:00 stdout F    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
2020-10-15T08:48:37.932096876+02:00 stdout F    at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
2020-10-15T08:48:37.932113131+02:00 stdout F    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
2020-10-15T08:48:37.932121817+02:00 stdout F    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
2020-10-15T08:48:37.932131116+02:00 stdout F    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:105) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]
2020-10-15T08:48:37.932140307+02:00 stdout F    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879) ~[spring-webmvc-5.2.7.RELEASE.jar!/:5.2.7.RELEASE]

Answer 1

根据流利的documentation：

其他步骤2：连接多行日志消息

应用程序日志存储在记录的log字段中。您可以使用fluent-plugin-concat过滤器来连接这些日志，然后再将其发送到目的地。
<filter docker.**>
  @type concat
  key log
  stream_identity_key container_id
  multiline_start_regexp /^-e:2:in `\/'/
  multiline_end_regexp /^-e:4:in/
</filter>

原始事件：

2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky","source":"stdout","log":"-e:2:in `/'"}
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"source":"stdout","log":"-e:2:in `do_division_by_zero'","container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky"}
2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"source":"stdout","log":"-e:4:in `<main>'","container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky"}

已过滤事件：

2016-04-13 14:45:55 +0900 docker.28cf38e21204: {"container_id":"28cf38e212042225f5f80a56fac08f34c8f0b235e738900c4e0abcf39253a702","container_name":"/romantic_dubinsky","source":"stdout","log":"-e:2:in `/'\n-e:2:in `do_division_by_zero'\n-e:4:in `<main>'"}

如果日志是典型的堆栈跟踪，请考虑使用 detect-exceptions 插件。

fluentbit将java stacktrace日志的每一行报告为新的日志事件

1 个答案: