Question

我在基于client-ip和包含日期+小时的时间戳创建指纹时遇到问题。我正在使用Logstash 7.3.1。这里是我的配置文件的相关部分

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date{ 
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  ...
  ruby{
    code => "
      keydate = Date.parse(event.get('timestamp'))
      event.set('keydate', keydate.strftime('%Y%m%d-%H'))
    "
  }
  fingerprint {
    key => "my_custom_secret"
    method => "SHA256"
    concatenate_sources => "true"
    source => [
      "clientip",
      "keydate"
    ]
  }
}

问题出在'ruby'块中。我尝试了多种方法来计算关键日期，但是没有给我带来错误的方法都无效。最后一个（使用此配置文件）是

[ERROR][logstash.filters.ruby    ] Ruby exception occurred: Missing Converter handling for full class name=org.jruby.ext.date.RubyDateTime, simple name=RubyDateTime

输入文档

{
      "timestamp" => "19/Sep/2019:00:07:56 +0200",
       "referrer" => "-",
       "@version" => "1",
     "@timestamp" => 2019-09-18T22:07:56.000Z,
             ...
        "request" => "index.php",
           "type" => "apache_access",
       "clientip" => "54.157.XXX.XXX",
           "verb" => "GET",
             ...
           "tags" => [
              [0] "_rubyexception"  # generated by the ruby exception above
           ],
       "response" => "200"
}

预期输出

{
      "timestamp" => "19/Sep/2019:00:07:56 +0200",
       "referrer" => "-",
       "@version" => "1",
     "@timestamp" => 2019-09-18T22:07:56.000Z,
             ...
        "request" => "index.php",
           "type" => "apache_access",
       "clientip" => "54.157.XXX.XXX",
           "verb" => "GET",
             ...
        "keydate" => "20190919-00", #format : YYYYMMDD-HH
    "fingerprint" => "ab347766ef....1190af",
       "response" => "200"
}

一如既往，非常感谢您的所有帮助！

Answer 1

我建议删除红宝石代码段，并使用内置的日期过滤器：https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html

您在ruby代码段中所做的正是日期过滤器的工作-从字段中提取时间戳并将其重构为您想要的格式。

另一个选择（不建议使用，但也可以使用）是使用grok来提取时间戳的相关部分并以不同的方式组合它们。

logstash：从时间戳记部分创建指纹

1 个答案: