Question

我是Logstash和Grok过滤器的新手。我想解析这些日志 -

2018-01-11 17:17:16,071 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | CommittedVirtualMemorySize :: 401186816 
2018-01-11 17:17:16,071 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | FreePhysicalMemorySize :: 1751130112 
2018-01-11 17:17:16,072 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | FreeSwapSpaceSize :: 4294967295 
2018-01-11 17:17:16,694 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | ProcessCpuLoad :: -1.0 
2018-01-11 17:17:16,694 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | ProcessCpuTime :: 47471104300 
2018-01-11 17:17:16,698 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | SystemCpuLoad :: 1.0 
2018-01-11 17:17:16,698 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | TotalPhysicalMemorySize :: 4285849600 
2018-01-11 17:17:16,698 | DEBUG | [Thread-2] | com.example.monitor.MonitorHelper:cpuMonitoring(307) | TotalSwapSpaceSize :: 4294967295

到像这样的JSON对象 -

{
  "timestamp": "2018-01-11 17:17:16,071",
  "log_level": "DEBUG",
  "thread_name": "Thread-2",
  "class": "com.example.monitor.MonitorHelper",
  "method": "cpuMonitoring",
  "line_number": "307",
  "CommittedVirtualMemorySize": "401186816",
  "FreePhysicalMemorySize": "1751130112",
  "FreeSwapSpaceSize": "4294967295",
  "ProcessCpuLoad": "-1.0",
  "ProcessCpuTime": "47471104300",
  "SystemCpuLoad": "1.0",
  "TotalPhysicalMemorySize": "4285849600",
  "TotalSwapSpaceSize": "4294967295"
}

截至目前，我的格言模式是 -

％{TIMESTAMP_ISO8601：timestamp} \ | ％{LOGLEVEL：log_level} \ | [（？\ b [\ w - ] + \ b）] \ | ％{JAVAFILE：class}：％{JAVAMETHOD：method}（％{NUMBER：line_number}）\ | ％{GREEDYDATA：log_message}

为每个输入日志行提供多个输出行。 JSON对象看起来像这样 -

{
  "timestamp": "2018-01-11 17:17:16,071",  
  "log_level": "DEBUG",
  "thread_name": "Thread-2",
  "class": "com.example.monitor.MonitorHelper",
  "method": "cpuMonitoring",
  "line_number": "307",
  "log_message": "CommittedVirtualMemorySize :: 401186816 "
}

为了达到这个目的，你能帮助我找到我需要的东西吗？

Answer 1

第一个建议是将原始日志输出更改为单行。

如果您无法使用filebeat发送文件，请使用FB multiline config合并这些行，然后再将其发送到logstash。

如果您不使用filebeat，可以尝试在logstash中使用multiline codec。

如何从logstash过滤器中的多个日志行生成单个输出json对象？

1 个答案: