Question

我第一次使用logstash，我正在尝试从amavisd-new中获取JSON报告以进行搜索和分析。 Amavisd-new能够将json日志写入redis，并且我完全导入了所有内容，并开始学习所有这些。

但是我有一个问题 - 来自amavis的JSON报告的格式如下所示 - 请注意“收件人”有一个数组，每个收件人都有一个条目。

我想将整个事件拆分为两个 - 每个收件人一个，留下所有其他字段相同，但替换每个收件人数组成员的“action”，“ccat_main”，“queued_as”等字段进入主要的甚至。

这个想法是，一个有两个收件人的传入事件会在logstash中产生两个单独的日志事件 - 每个人一个。

我看过分裂事件，但我没有看到如何做到这一点 - 我似乎无法在任何地方找到任何合适的例子。

因此，对于真实的单词示例，请给出：

 {
    "@timestamp" => "2014-05-06T09:29:47.048Z",
    "time_unix" => 1399368587.048,
    "time_iso_week_date" => "2014-W19-2",
    "partition" => "19",
    "type" => "amavis",
    "host" => "mailer.example.net",
    "queued_as" => ["3gNFyR4Mfjzc3", "3gNFyR4n6Lzc4"],
    "recipients" => [
      { "action" => "PASS",
        "ccat_main" => "Clean",
        "queued_as" => "3gNFyR4Mfjzc3",
        "rcpt_is_local" => false,
        "rcpt_to" => "recip2@example.org",
        "smtp_code" => "250",
        "smtp_response" => "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4Mfjzc3",
        "spam_score" => -2.0
      },
      { "action" => "PASS",
        "ccat_main" => "Clean",
        "mail_id_related" => "men7HTERZaOF",
        "penpals_age" => 1114599,
        "queued_as" => "3gNFyR4n6Lzc4",
        "rcpt_is_local" => true,
        "rcpt_to" => "recip1@example.net",
        "smtp_code" => "250",
        "smtp_response" => "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4n6Lzc4",
        "spam_score" => -5.272
      }
    ],
    "smtp_code"  => ["250"],
  }

我想结束两个不同的事件，比如：

  {
    "@timestamp" => "2014-05-06T09:29:47.048Z",
    "time_unix" => 1399368587.048,
    "time_iso_week_date" => "2014-W19-2",
    "partition" => "19",
    "type" => "amavis",
    "host" => "mailer.example.net",
    "queued_as" => ["3gNFyR4Mfjzc3", "3gNFyR4n6Lzc4"],
    "action" => "PASS",
    "ccat_main" => "Clean",
    "queued_as" => "3gNFyR4Mfjzc3",
    "rcpt_is_local" => false,
    "rcpt_to" => "recip2@example.org",
    "smtp_code" => "250",
    "smtp_response" => "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4Mfjzc3",
    "spam_score" => -2.0
    "smtp_code"  => ["250"],
  }

和

  {
    "@timestamp" => "2014-05-06T09:29:47.048Z",
    "time_unix" => 1399368587.048,
    "time_iso_week_date" => "2014-W19-2",
    "partition" => "19",
    "type" => "amavis",
    "host" => "mailer.example.net",
    "queued_as" => ["3gNFyR4Mfjzc3", "3gNFyR4n6Lzc4"],
    "recipients" => [
    "action" => "PASS",
    "ccat_main" => "Clean",
    "mail_id_related" => "men7HTERZaOF",
    "penpals_age" => 1114599,
    "queued_as" => "3gNFyR4n6Lzc4",
    "rcpt_is_local" => true,
    "rcpt_to" => "recip1@example.net",
    "smtp_code" => "250",
    "smtp_response" => "250 2.0.0 from MTA(smtp:[::1]:10013): 250 2.0.0 Ok: queued as 3gNFyR4n6Lzc4",
    "spam_score" => -5.272
    "smtp_code"  => ["250"],
  }

修改

好的，我只是使用了拆分过滤器 - 我应该已经看过了。但有一件事令我困惑。

当有一个收件人时，它会直接通过该块 - kibana的结果如下：

recipients      {
  "action": "PASS",
  "bypass_banned_checks": true,
  "bypass_spam_checks": true,
  "ccat_main": "Clean",
  "queued_as": "3qv7Km4Ybpz14Kyh",
  "rcpt_is_local": true,
  "rcpt_to": "user@domain.com",
  "rid": "552213780",
  "smtp_code": "250",
  "smtp_response": "250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 3qv7Km4Ybpz14Kyh"
}

但是当有2个或更多收件人时，新事件看起来都是这样的，并附有相应的信息：

recipients.action       PASS
recipients.ccat_main        CleanTag
recipients.queued_as        3qv7Ly4Pqvz4wyS
recipients.rcpt_is_local        true
recipients.rcpt_to      user@domain.com
recipients.rid      552278239
recipients.smtp_code        250
recipients.smtp_response        250 2.0.0 from MTA(smtp:[127.0.0.1]:10025): 250 2.0.0 Ok: queued as 3qv7Ly4Pqvz4wyS
recipients.whitelisted      true

为什么两者有区别？我想我更喜欢将收件人字段保留为值的哈希值，那么使拆分事件与单个事件保持一致的最佳方法是什么？

Answer 1

那是split filter。在每个副本中，您可以将字段重命名为正确的级别，或者删除每个副本中不需要的字段。

Answer 2

这就是我最终做的事情。这使得它是否一致是否有一个或多个数组成员被拆分。

这可能是一种更简单的方法，但现在这已经覆盖了我。如果我拿出别的东西，我会回来修改。

filter {
    split {
        field => "recipients"
        target => "recipcopy"
        remove_field => "recipients"
    }
}

filter {
    if [recipients] {
        ruby {
            code => "event['recipcopy'] = event['recipients'][0]"
            remove_field => "recipients"
        }
    }
}

filter {
    if [recipcopy] {
        mutate {
            rename => { "recipcopy" => "recipients" }
        }
    }
}

logstash - 根据包含的数组将事件拆分为两个

2 个答案: