Question

iam通过logstash发送此日志

2017-02-27T13:00:07+01:00    test    {"createdAt":"2017-02-27T13:00:07+0100","cluster":"undefined","nodeName":"undefined","nodeIP":"10.11.11.50","clientIP":"10.11.11.72","customerId":1,"identityId":332,"appType":"admin","eventGroup":"education","eventName":"insert","eventData":{"education_insert":{"type":"course","data":{"education_id":2055,"education":{"id":2055,"customer_id":1,"creator_id":332,"type":"course","status":"new","is_featured":false,"enroll_deadline":null,"complete_deadline":null,"count_view":0,"count_like":0,"meta_title":"test Course - progress","meta_description":"test Course - progress","discoverable":"everyone","progress_max":0,"instructor_ids":[332],"tag_ids":[135],"discoverable_group_ids":[],"category_ids":[14],"audits":null,"instructors":null,"creator":null,"lessonGroups":null,"categories":null},"duration":"quick"}}},"scopeType":"education","scopeId":"2055"}

如何删除2017-02-27T13:00:07+01:00和test.app.event

Answer 1

您需要使用grok提取消息的json部分，然后使用json过滤器将提取的json转换为事件。最后，您需要删除在最终事件中不需要的mutate字段（例如message）。

Answer 2

您可以使用正则表达式模式，以便只获取json。 regex模式应位于模式文件中：

模式可能如下：

REQUIREDDATA {([^}]*)}([^}]*)([^}]*)}}}([^}]*)} <-- this extracts only your json part

经过测试的正则表达式pattern。

此后您可以在 grok 匹配中使用该模式：

grok {
    patterns_dir => ["/pathto/patterns"]
    match => { "message" => "^%{REQUIREDDATA:new}" }            
}

现在，该消息只有日志行中的JSON部分，因此您现在可以通过logstash将其推送到ES。以上只是一个示例，以便您可以重现。

希望它有所帮助！

Answer 3

我用过这个并为我工作:) thx求助

ingest-attachment

Logstash只发送json

3 个答案: