我目前正在使用日志,其中一些内容如下所示:
00:19:59.771 (07120/evtThread ) TRC> Cem< [Core1] CALL_STATE...
00:20:00.199 (05768/BCMApplicationThread) INF>
#S#|Call stats, ongoing calls: 8, handled_calls: 7304
#S#+----------------------------+----------+----------+----------+----------+----------+
#S#|Peer | From| To| MinTime| MaxTime| AvgTime|
#S#+----------------------------+----------+----------+----------+----------+----------+
#S#| CallDispatcher:Core2| 0| 0| 0| 0| 0|
#S#| CallDispatcher:Core3| 0| 0| 0| 0| 0|
#S#| Cem:Core1| 1632| 6207| 0| 5996522| 311685|
我解析了包含时间的行:
grok {
match => [ "message", "%{TIME:time} (?<bcm_comp>\(\d{5}\/\w{4,}\:*\ *\w*\)) (?<loglevel>\w{3}>{1}) %{GREEDYDATA:message}" ]
overwrite => [ "message" ]
add_field => [ "BCM_System", "PROD" ]
}
前面包含#S#的行被解析了。忽略包含--------的行和包含表名称和调用统计行的行。
grok {
match => [ "message", "(?<start>\#\S\#\|)\s* (?<peer>\w*\:\w*)(?<div2>\|)\s* %{NUMBER:From}(?<div3>\|)\s* %{NUMBER:To}(?<div4>\|)\s* %{NUMBER:MinTime}(?<div5>\|)\s* %{NUMBER:MaxTime}(?<div6>\|)\s* %{NUMBER:AvgTime}(?<div7>\|)" ]
remove_field => [ "start", "div2", "div3", "div4", "div5", "div6", "div7" ]
overwrite => [ "message"]
add_field => [ "reference_time", "%{@time}"]
}
我要做的是从前一行开始,将其添加为我在#s#行中搜索的字段。我尝试使用logstash中的add_field语法,如图所示,但它不起作用......它只是打印出%{@ time}。
有什么方法可以以某种方式从前一行中提取时间并将其放在另一个事件的字段中?
答案 0 :(得分:1)
据我所知,您必须编写一个过滤插件来执行此类操作。这是一个简单的插件,我把它拼凑在一起做类似的事情 - 当它看到它时会记住一个字段,然后使用它看到的最后一个值,如果它不存在的话。
# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"
require "set"
#
# This filter will look for a field from an event and record the last value
# of it. If it's not present, it will add the last value to the event
#
# The config looks like this:
#
# filter {
# memorize {
# field => "time"
# default => "00:00:00.000"
# }
# }
#
# The `field` is the name of the field that you want to memorize
# The `default` is the value to use for the field if you haven't seen it yet
# in the file (this is optional)
class LogStash::Filters::Memorize < LogStash::Filters::Base
config_name "memorize"
milestone 1
# The field to memorize
config :field, :validate => :string, :required => true
# the default value to use for the field if it's not seen before we need it
config :default, :validate => :string, :required => false
# The stream identity is how the multiline filter determines which stream an
# event belongs to. See the multiline plugin if you want more details on how
# this might work
config :stream_identity , :validate => :string, :default => "%{host}.%{path}.%{type}"
public
def initialize(config = {})
super
@threadsafe = false
# This filter needs to keep state.
@memorized = Hash.new
end # def initialize
public
def register
# nothing needed
end # def register
public
def filter(event)
return unless filter?(event)
if event[@field].nil?
val = @memorized[@stream_identity]
if val.nil?
val = @default
end
event[@field] = val
filter_matched(event)
else
@memorized[@stream_identity] = event[@field]
end
end
end