Question

您好我使用filebeat和logstash解析kubernetes容器多行有些问题。 kubernetes日志文件位于/var/log/containers/*.log和json行结构中。

我的配置有什么问题吗？我错过了什么？

filebeat.yml：

filebeat:
  # List of prospectors to fetch data.
  prospectors:
    -
      paths:
        - /var/log/containers/*.log
      fields: {log_type: containers}
      ignore_older: 5m
      symlinks: true
      json.message_key: log
      json.keys_under_root: true
      json.add_error_key: true
      multiline.pattern: '^\d{4}-\d{2}-\d{2}'
      multiline.match: after
      multiline.negate: true
      document_type: kube-logs
  registry_file: /var/log/containers/filebeat_registry
output:
  logstash:
    hosts: ["logstash-logging:5044"]

logstash.conf：

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "kube-logs" {

    date {
      match => ["time", "ISO8601"]
      remove_field => ["time"]
    }
    json {
      source => "message"
    }

    grok {
      match => [ "log", "<SOME_PATTERN>" ]
      overwrite => [ "message" ]
    }

}

kubernetes容器：

{"log":"11:11:17,740 |-INFO in ch.qos.logback.core.joran.action.mapWAR - Attaching appender named [FILE-LOG] to Logger[ROOT]\n","stream":"stdout","time":"2017-05-09T11:11:17.742837362Z"}
{"log":"11:11:17,740 |-INFO in ch.qos.logback.classic.joran.action.mapWAR - End of configuration.\n","stream":"stdout","time":"2017-05-09T11:11:17.742840277Z"}
{"log":"11:11:17,741 |-INFO in ch.qos.logback.classic.joran.mapWAR - Registering current configuration as safe fallback point\n","stream":"stdout","time":"2017-05-09T11:11:17.742843277Z"}
{"log":"\n another line","stream":"stdout","time":"2017-05-09T11:11:17.742846485Z"}
{"log":"09-May-2017 11:11:17.756 INFO [localhost-startStop-1] org.apache.catalina.startup.mapWAR nice","stream":"stderr","time":"2017-05-09T11:11:17.756924376Z"}
{"log":"09-May-2017 11:11:17.757 INFO [localhost-startStop-1] org.apache.catalina.startup.mapWAR great","stream":"stderr","time":"2017-05-09T11:11:17.757465828Z"}

Answer 1

我认为你仍然需要把这条线放在一起，你能尝试一下吗？使用{因为日志以{不是您的时间戳格式开头。

filebeat.prospectors:
- paths:
    - input.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after

processors:
- decode_json_fields:
    fields: ['message']
    target: json

output.console.pretty: true

Filebeat multiline kubernetes容器日志无法正常工作

1 个答案: