Question

我正在尝试根据进程ID汇总事件。进程ID顺序不正确，跨多行。下面是一个这样的示例-

2019 Apr 23 14:01:44:870 GMT +0000 BW.EnterpriseFlifoFormatter-EnterpriseFlifoFormatter USER [BW-User] - Interface Name : EFF: Process ID :674526635 Main Process has started^M
2019 Apr 23 14:01:44:870 GMT +0000 BW.EnterpriseFlifoFormatter-EnterpriseFlifoFormatter USER [BW-User] - Interface Name : EFF: Process ID :674526636 Main Process has started^M
2019 Apr 23 14:01:44:878 GMT +0000 BW.EnterpriseFlifoFormatter-EnterpriseFlifoFormatter USER [BW-User] - Interface Name : EFF: Process ID :674526636 Ending ODS query Process to create enterprise message^M
2019 Apr 23 14:01:44:882 GMT +0000 BW.EnterpriseFlifoFormatter-EnterpriseFlifoFormatter USER [BW-User] - Interface Name : EFF: Process ID :674526635 Send Message Process has finished at  :  1556028104882 for flight 1206 departing on 2019-04-24 from EWR to CLT of type Leg Level Message^M
2019 Apr 23 14:01:44:882 GMT +0000 BW.EnterpriseFlifoFormatter-EnterpriseFlifoFormatter USER [BW-User] - Interface Name : EFF: Process ID :674526635 Send Message Process has finished at  :  1556028104882 for flight 1206 departing on 2019-04-24 from EWR to CLT of type Flight Level Message^M
2019 Apr 23 14:01:44:882 GMT +0000 BW.EnterpriseFlifoFormatter-EnterpriseFlifoFormatter USER [BW-User] - Interface Name : EFF: Process ID :674526636 Send Message Process has begun at  :  1556028104882 for flight 1196 departing on 2019-04-24 from CUN to ORD of type Leg Level Message^M
2019 Apr 23 14:01:44:883 GMT +0000 BW.EnterpriseFlifoFormatter-EnterpriseFlifoFormatter USER [BW-User] - Interface Name : EFF: Process ID :674526636 Send Message Process has begun at  :  1556028104882 for flight 1196 departing on 2019-04-24 from CUN to ORD of type Flight Level Message^M
2019 Apr 23 14:01:44:882 GMT +0000 BW.EnterpriseFlifoFormatter-EnterpriseFlifoFormatter USER [BW-User] - Interface Name : EFF: Process ID :674526635 Main Process is ending for flight 1206 departing on 2019-04-24^M
2019 Apr 23 14:01:44:882 GMT +0000 BW.EnterpriseFlifoFormatter-EnterpriseFlifoFormatter USER [BW-User] - Interface Name : EFF: Process ID :674526636 Main Process is ending for flight 1206 departing on 2019-04-24^M

因此，进程ID并不是按顺序排列的，我们如何使用Logstash聚合过滤器来关联并发送属于同一进程ID的所有消息作为单个事件

Answer 1

我建议您检查此plugin来跟踪一对开始/结束事件，并使用它们的时间戳来计算它们之间的经过时间，因此，根据需要，您可以根据ID标识一个“开始”事件，并等待其相关的“结束”事件。这是使用它的简单example。

调整此插件以执行所需的操作可能很有用，因为它已处理超时，等待基于特定字段的事件等。

但是您还需要注意以下几点：

知道何时停止，直到何时需要累积事件？
可以在内存中累积很多事件吗？
何时超时？

如果要摄取大量数据，对我来说Logstash似乎不太适合这种需求。考虑使用其他工具汇总事件，这可能是apache spark的日常工作，可以使用Elasticsearch作为后端汇总事件？

编辑：我迅速检查了插件的source，基本上，我的建议是将所需的所有信息添加到“统一事件”中。

假设您需要从所有事件中获取“ my_field”的内容。您可以更新第167行：

test1

最后编辑：这个插件应该适合你 https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html

Logstash聚合插件，其任务ID跨多行

1 个答案: