Question

我的目标是在logstash中基于pId组合事件。但我发现具有相同pId的事件不会合并为一个事件。添加聚合后，我无法看到任何更改。请帮助

日志看起来像这样：

June 1st 2017, 11:51:26.992 {id} {pId} ClassName:methodName:99 [DEBUG] - Received request:
June 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - Id: abbababcajdfbjasndflsdlf
June 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - unique id: AAAAA
June 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] Total time: 12

这是我的配置：

filter {
grok{
match => { "message" => "%{DATESTAMP:log_timestamp} %{DATA:id} %{DATA:pId} %{DATA:ClassName} [%{LOGLEVEL:severity}] - %{GREEDYDATA:message}" }
}
if [message] =~ /Received request:/ {
aggregate {
task_id => "%{pId}"
code => "map['message'] = event['message']"
map_action => "create"
}
}
else if [message] =~ /Total time:^/ {
aggregate {
task_id => "%{pId}"
code => "map['new_message'] = event['message'];event['new_message'] = map['new_message']"
map_action => "update"
end_of_task => true
timeout => 120
}
}
else {
aggregate {
task_id => "%{pId}"
code => "map['new_message'] = event['message'];event['new_message'] = map['new_message']"
map_action => "update"
}
}
}

Answer 1

Aggregate是其中一个可能真的难以正确使用的过滤器。在很大程度上，因为Logstash是从螺栓设计为并行处理管道，所以过滤器堆栈中的每个aggregate调用对于管道是唯一的，并且您无法确定是否将运行所有事件通过相同的管道。开箱即用，就是这样。

如果使用-w 1参数运行logstash以强制所有内容通过单个管道，则会出现此行为。

在这种情况下，我建议改为使用multiline上的input编解码器。这会将所有日志整合在一个事件中，您可以稍后在过滤器阶段进行分析。当然，这假设这些多行事件中的每一个都同时被丢弃并且不会被多路复用。如果你得到多路复用，那么聚合将需要失去你的并行性。

input {
  file {
    path => "/var/log/app/debug_logs.log"
    codec => multiline {
       pattern => "Received request:"
       negate => true
       what => previous
    }
  }
}

这将搜索不匹配您的Received request:正则表达式的事件，并将它们附加到上一行。当它看到Received request:时，它将启动一个新事件。您的filter {}阶段将会看到此

message => "June 1st 2017, 11:51:26.992 {id} {pId} ClassName:methodName:99 [DEBUG] - Received request:\nJune 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - Id: abbababcajdfbjasndflsdlf\nJune 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - unique id: AAAAA\nJune 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] Total time: 12"

在并行环境中操作更容易。

聚合logstash过滤器配置

1 个答案: