Question

Filebeat 在向 Logstash 发送日志时会定期挂起，调试日志不会显示任何错误，它只是一直显示文件末尾到达的日志，但实际上并没有读取任何日志并将它们发送到 Logstash。 filebeat 和 Logstash 都有足够的资源。只有重新启动 filebeat 或 logstash 才能解决问题。有没有人遇到过同样的错误？下面是 Filebeat 和 Logstash 的调试日志

我们在每个 kubernetes 节点上将 filebeat 作为 deamonset 运行，并将 logstash 作为 kubernetes 服务运行，并为两者使用弹性舵图

filebeat 和 logstash 版本 - 7.9.3

Filebeat 日志

2021-02-17T06:29:59.088Z    DEBUG   [harvester] log/log.go:107  End of file reached: /var/lib/docker/containers/2130e05/2130e05-json.log; Backoff now.
2021-02-17T06:29:59.259Z    DEBUG   [harvester] log/log.go:107  End of file reached: /var/lib/docker/containers/232bc67/232bc67-json.log; Backoff now.
2021-02-17T06:29:59.340Z    DEBUG   [harvester] log/log.go:107  End of file reached: /var/lib/docker/containers/65c977797/65c977797-json.log; Backoff now.
2021-02-17T06:29:59.550Z    DEBUG   [harvester] log/log.go:107  End of file reached: /var/lib/docker/containers/613a433de/613a433de-json.log; Backoff now.
2021-02-17T06:29:59.945Z    DEBUG   [harvester] log/log.go:107  End of file reached: /var/lib/docker/containers/fafdaa9cfdc3/fafdaa9cfdc3-json.log; Backoff now.

Logstash 日志

[2021-02-05T07:05:50,780][DEBUG][org.logstash.beats.ConnectionHandler][main] 884460aa: batches pending: true
[2021-02-05T07:05:53,405][DEBUG][org.logstash.beats.ConnectionHandler][main] 4f6b0df8: reader and writer are idle, closing remote connection
[2021-02-05T07:05:53,475][DEBUG][org.logstash.beats.ConnectionHandler][main] 17aaab9e: batches pending: true
[2021-02-05T07:05:54,110][DEBUG][org.logstash.beats.ConnectionHandler][main] 8cff884d: batches pending: true
[2021-02-05T07:06:15,369][DEBUG][org.logstash.beats.ConnectionHandler][main] babac4ef: batches pending: true
[2021-02-05T07:06:21,704][DEBUG][org.logstash.beats.ConnectionHandler][main] b8f4c8f7: batches pending: true
[2021-02-05T07:06:28,794][DEBUG][org.logstash.beats.ConnectionHandler][main] a37e2da3: batches pending: true
[2021-02-05T07:06:29,834][DEBUG][org.logstash.beats.ConnectionHandler][main] 455c7c87: batches pending: true
[2021-02-05T07:06:29,834][DEBUG][org.logstash.beats.ConnectionHandler][main] 455c7c87: batches pending: true
[2021-02-05T07:06:30,794][DEBUG][org.logstash.beats.ConnectionHandler][main] a895e7d3: batches pending: true
[2021-02-05T07:06:36,704][DEBUG][org.logstash.beats.ConnectionHandler][main] b8f4c8f7: batches pending: true
[2021-02-05T07:06:38,476][DEBUG][org.logstash.beats.ConnectionHandler][main] 17aaab9e: batches pending: true
[2021-02-05T07:06:42,113][DEBUG][org.logstash.beats.ConnectionHandler][main] 8cff884d: batches pending: true
[2021-02-05T07:06:50,779][DEBUG][org.logstash.beats.ConnectionHandler][main] b5569bb1: reader and writer are idle, closing remote connection
[2021-02-05T07:06:50,779][DEBUG][org.logstash.beats.ConnectionHandler][main] 6997090c: reader and writer are idle, closing remote connection

文件节拍配置

filebeatConfig:
  filebeat.yml: |
    filebeat.inputs:
    - type: container
      paths:
        - /var/lib/docker/containers/*/*.log
      multiline.pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{3}\s'
      multiline.negate: true
      multiline.match: after
      multiline.max_lines: 2000
      close_inactive: 24h
      processors:
      - add_kubernetes_metadata:
          host: ${NODE_NAME}
          matchers:
          - logs_path:
              logs_path: "/var/lib/docker/containers/"
      - drop_fields:
          fields: ["kubernetes.labels.app"]
      - drop_event:
           when:
            or:
             - equals:
                 kubernetes.namespace: "logging"
             
    queue.mem:
      events: 131072         
    logging.level: debug
    output.file.enabled: false
    output.logstash:
        hosts:
        - logstash:5044
        loadbalance: true
        compression_level: 3
        worker: 10
        bulk_max_size: 8192
        pipelining: 0
        ttl: 120s

Filebeat 会定期挂起并停止向 logstash 发送日志

0 个答案: