Question

我有一个简单的架构，我的 Java 应用程序日志被“FileBeat”拉取，beats 将日志推送到“Elastic Search”，如下所示

java App ---> FileBeat ---> Elastic-Search ---> Kibana

在我的情况下，最初 File-beat 成功地将日志推送到 Elastic 搜索，而 kibana 正在获取日志并显示，但一段时间后，日志没有推送到 Elastic-Search ，我必须重新启动 Filebeat 才能再次看到 Kibana 中滚动的日志。这种情况发生了多次

FileBeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/ubuntu/workspace/logs/*.log
      #json.message_key: log
      #json.keys_under_root: true
  #processors:
  #- add_docker_metadata: ~
  #multiline.type: pattern
  #multiline.pattern: '^\[0-9]{4}-[0-9]{2}-[0-9]{2}'
  #multiline.negate: true
  #multiline.match: after
output.elasticsearch:
  hosts: ["<**elastic-aws-url**>"]
setup.ilm.enabled: false
setup.pack.security.enabled: false
setup.xpack.graph.enabled: false
setup.xpack.watcher.enabled: false
setup.xpack.monitoring.enabled: false
setup.xpack.reporting.enabled: false

弹性堆栈版本：

弹性搜索 7.8.0
filebeat-oss:7.8.0
kibana 7.8.0

filebeat 在他不向 ES 发送日志时记录；

2021-05-18T16:08:53.334Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1500,"time":{"ms":5}},"total":{"ticks":4160,"time":{"ms":5},"value":4160},"user":{"ticks":2660}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"274fa123-69a9-4a02-a662-34da22b86f92","uptime":{"ms":900022}},"memstats":{"gc_next":17130176,"memory_alloc":9042752,"memory_total":250665632},"runtime":{"goroutines":19}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.08,"15":0.17,"5":0.17,"norm":{"1":0.01,"15":0.0213,"5":0.0213}}}}}}
2021-05-18T16:09:23.334Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1510,"time":{"ms":6}},"total":{"ticks":4170,"time":{"ms":6},"value":4170},"user":{"ticks":2660}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"274fa123-69a9-4a02-a662-34da22b86f92","uptime":{"ms":930022}},"memstats":{"gc_next":17130176,"memory_alloc":9390752,"memory_total":251013632},"runtime":{"goroutines":19}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.05,"15":0.16,"5":0.16,"norm":{"1":0.0063,"15":0.02,"5":0.02}}}}}}
2021-05-18T16:09:53.333Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1520,"time":{"ms":6}},"total":{"ticks":4180,"time":{"ms":6},"value":4180},"user":{"ticks":2660}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"274fa123-69a9-4a02-a662-34da22b86f92","uptime":{"ms":960022}},"memstats":{"gc_next":17130176,"memory_alloc":9647656,"memory_total":251270536},"runtime":{"goroutines":19}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.11,"15":0.16,"5":0.16,"norm":{"1":0.0138,"15":0.02,"5":0.02}}}}}}
2021-05-18T16:10:23.333Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1530,"time":{"ms":15}},"total":{"ticks":4190,"time":{"ms":15},"value":4190},"user":{"ticks":2660}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"274fa123-69a9-4a02-a662-34da22b86f92","uptime":{"ms":990023}},"memstats":{"gc_next":17129728,"memory_alloc":8597232,"memory_total":251617832},"runtime":{"goroutines":19}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.13,"15":0.16,"5":0.16,"norm":{"1":0.0163,"15":0.02,"5":0.02}}}}}}
2021-05-18T16:10:53.333Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1540,"time":{"ms":5}},"total":{"ticks":4200,"time":{"ms":5},"value":4200},"user":{"ticks":2660}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"274fa123-69a9-4a02-a662-34da22b86f92","uptime":{"ms":1020022}},"memstats":{"gc_next":17129728,"memory_alloc":8955896,"memory_total":251976496},"runtime":{"goroutines":19}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.18,"15":0.17,"5":0.17,"norm":{"1":0.0225,"15":0.0213,"5":0.0213}}}}}}
2021-05-18T16:11:23.334Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1540,"time":{"ms":7}},"total":{"ticks":4200,"time":{"ms":7},"value":4200},"user":{"ticks":2660}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"274fa123-69a9-4a02-a662-34da22b86f92","uptime":{"ms":1050022}},"memstats":{"gc_next":17129728,"memory_alloc":9398688,"memory_total":252419288},"runtime":{"goroutines":19}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.24,"15":0.18,"5":0.18,"norm":{"1":0.03,"15":0.0225,"5":0.0225}}}}}}

对此行为的任何见解都表示赞赏..谢谢

filebeat 在一段时间后停止将日志推送到 Elastic 搜索

0 个答案: