Question

我正在编写一个logstash 2.4.0配置来通过HTTP日志。我们希望在Header字段中传递的PORT包含在下面的Line字段中。没有定义特定的最终事件。虽然我也尝试过添加结束事件。

我目前使用的输入日志文件是：

HEADER 9200
LINE 1 2016-10-05 08:39:00 Some log data
LINE 2 2016-10-05 08:40:00 Some other log data
FOOTER
HEADER 9300
LINE 4 2016-11-05 08:39:00 Some log data in another log
LINE 5 2016-11-05 08:40:00 Some other log data in another log
FOOTER

我想有这样的输出：输出

中当前缺少 Server_port 字段

{"message" => "HEADER 9200",
 "@version" => "1",
 "@timestamp" => "2016-11-15T11:17:18.425Z",
 "path" => "test.log",
 "host" => "hostname",
 "type" => "event",
 "env" => "test",
 "port" => 9200,
 "tags" => [[0] "Header"]    }
{"message" => "LINE 1 2016-10-05 08:39:00 Some log data",
 "@version" => "1",
 "@timestamp" => "2016-11-15T11:17:20.186Z",
 "path" => "test.log",
 "host" => "hostname",
 "type" => "event",
 "env" => "test",
 "logMessage" => "1 2016-10-05 08:39:00 Some log data",
 "Server_port" => 9200,
 "tags" => [[0] "Line"]}
{"message" => "LINE 2 2016-10-05 08:40:00 Some other log data",
 "@version" => "1",<
 "@timestamp" => "2016-11-15T11:17:20.192Z",
 "path" => "test.log",
 "host" => "hostname",
 "type" => "event",
 "env" => "test",
 "logMessage" => "2 2016-10-05 08:40:00 Some other log data",
 "Server_port" => 9200,
 "tags" => [[0] "Line"]}
{"message" => "FOOTER",
 "@version" => "1",
 "@timestamp" => "2016-11-15T11:17:20.195Z",
 "path" => "test.log",
 "host" => "hostname",
 "type" => "event",
 "env" => "test",
 "tags" => [[0] "Footer"]}

尝试不同的东西之后，我正在使用的配置如下，使用硬编码的taskid ='abcd'进行测试：

input{ file{    path => "test.log"
                start_position => "beginning"
                sincedb_path => "/dev/null"
                ignore_older => 0
                type => "event"
                add_field => { "env" => "test"} }
}
filter{
        grok {
                break_on_match => false
                tag_on_failure => []
                match => {"message" => ["^HEADER%{SPACE}%{INT:port:int}"]}
                add_tag => ["Header"]
                }
        grok {
                break_on_match => false
                tag_on_failure => []
                match => {"message" => "^LINE%{SPACE}%{GREEDYDATA:logMessage}"}
                add_tag => ["Line"]
                }
        grok {
                break_on_match => false
                tag_on_failure => []
                match => {"message" => "^FOOTER"}
                add_tag => ["Footer"]
                }    
       if "Header" in [tags]{
                aggregate{
                        task_id => "abcd"
                        code => "map['server_port'] ||= 0; map['server_port']=event['port']"
                        push_map_as_event_on_timeout => true
                        push_previous_map_as_event => true
                        map_action => "create"
                }
        }
        elseif "Line" in [tags]{
                aggregate{
                        task_id => "abcd"
                        code => "event.set('server_port',map['server_port'])"                                                   
                        map_action => "update"
                }
        }
        else if "Footer" in [tags]{
                aggregate{
                        task_id => "abcd"
                        code => "event.set('server_port',map['server_port'])"                                                   
                        map_action => "update"
                        end_of_task => true
                        timeout => 120
                }
        }
}
output {
  stdout { codec => rubydebug }
}

虽然此配置运行时没有错误，但它不会创建server_port字段。我哪里错了？

Answer 1

在摆弄更多的东西后，我有一个工作测试用例。我已按如下方式更改了配置：

grok {
                break_on_match => false
                tag_on_failure => []
                match => {
                   "message" => ["^HEADER%{SPACE}%{INT:taskid:int}%{SPACE}%{INT:port:int}"]
                }
                add_tag => ["Header"]
                }

和

 if "Header" in [tags]{
            aggregate{
                    task_id => "%{taskid}"
                    code => "map['port']=event.get('port')"
                    map_action => "create"
            }
    }
    elseif "Line" in [tags]{
            aggregate{
                    task_id =>"%{taskid}"
                    code => "event.set('port',map['port'])"
                    map_action => "update"
            }
    }
    else if "Footer" in [tags]{
            aggregate{
                    task_id => "%{taskid}"
                    code => "event.set('port',map['port'])"
                    map_action => "update"
                    end_of_task => true
                    timeout => 120
            }
    }

并在日志中添加了一个任务ID字段：

HEADER 123 9200
LINE 123 2016-10-05 08:39:00 Some log data

Logstash聚合过滤器，将信息添加到下一行

1 个答案: