Question

我有一个Cellery应用，我们称之为通用myapp。日志配置为JSON格式。我们从与之合作的一些运营商那里收到一些条目，但不幸的是，这些条目实际上不是JSON格式的特定条目。我正在尝试创建一个logstash过滤器，以便可以将这些日志导入elasticsearch的特定索引myapp中。

摘录自日志：

"rpc_method": "get_comm_details", "rpc_params": [52683], "timestamp": "2018-10-16T20:23:24.165372Z"}
{"event": "Task succeeded", "id": "890a084e-5ff8-4d3f-aedb-2a8688bb62ff", "level": "info", "logger": "celery.worker", "name": "myapp.tasks.RPCTask", "rpc_endpoint": "user", "rpc_method": "get_comm_details", "rpc_params": [52683], "runtime": 0.3010744289495051, "service": "core", "timestamp": "2018-10-16T20:23:24.466367Z"}


{"event": "Received task", "id": "f1b5357d-5902-447a-9bd0-b69c5d0a416f", "level": "info", "logger": "celery.worker", "name": "myapp.tasks.RPCTask", "rpc_endpoint": "wallet.payment.request", "rpc_method": "get_incoming_requests", "rpc_params": [52683], "timestamp": "2018-10-16T20:23:25.307046Z"}

{"event": "Task accepted", "id": "f1b5357d-5902-447a-9bd0-b69c5d0a416f", "level": "info", "logger": "celery.worker", "name": "myapp.tasks.RPCTask", "pid": 140559562601968, "rpc_endpoint": "wallet.payment.request", "rpc_method": "get_incoming_requests", "rpc_params": [52683], "timestamp": "2018-10-16T20:23:25.308901Z"}

{"event": "Task succeeded", "id": "f1b5357d-5902-447a-9bd0-b69c5d0a416f", "level": "info", "logger": "celery.worker", "name": "myapp.tasks.RPCTask", "rpc_endpoint": "wallet.payment.request", "rpc_method": "get_incoming_requests", "rpc_params": [52683], "runtime": 0.016890593920834363, "service": "wallet", "timestamp": "2018-10-16T20:23:25.325567Z"}

{"event": "Received task", "id": "23fb661f-a256-488a-bf22-3680fe4d4f32", "level": "info", "logger": "celery.worker", "name": "myapp.tasks.SilentRPCTask", "rpc_endpoint": "analytics.fact", "rpc_method": "add", "rpc_params": {"city_id": null, "correlation_id": "6ad37ee2-e4a8-46ab-9dad-fb3d16918fcc", "country_id": null, "dst_is_platform": true, "dst_ln": 21580012, "dst_mid": null, "dst_operator_id": null, "dst_package_id": null, "dst_user_id": null, "duration": null, "ignore_errors": true, "medium": null, "platform_event": "sms-in", "service": "wallet", "service_event": null, "service_path": null, "sms_in_size": 1, "sms_out_size": null, "src_is_platform": false, "src_ln": 647234242, "src_mid": null, "src_operator_id": 300046, "src_package_id": null, "src_user_id": 313676, "ts_end": null, "ts_start": null}, "timestamp": "2018-10-16T20:23:27.739982Z"}

上面的日志已在Elasticsearch中正确索引为JSON。不幸的是，日志中也有如下几行：

{"event": "Task succeeded", "id": "f1b5357d-5902-447a-9bd0-b69c5d0a416f", "level": "info", "logger": "celery.worker", "name": "myapp.tasks.RPCTask", "rpc_endpoint": "wallet.payment.request", "rpc_method": "get_incoming_requests", "rpc_params": [52683], "runtime": 0.016890593920834363, "service": "wallet", "timestamp": "2018-10-16T20:23:25.325567Z"}
{"event": "Task succeeded", "id": "f1b5357d-5902-447a-9bd0-b69c5d0a416f", "level": "info", "logger": "celery.worker", "name": "myapp.tasks.RPCTask", "rpc_endpoint": "wallet.payment.request", "rpc_method": "get_incoming_requests", "rpc_params": [[77621]], "runtime": 0.016890593920834363, "service": "wallet", "timestamp": "2018-10-16T20:23:25.325567Z"}

并非JSON兼容格式。我应该使用哪种类型的过滤器，以便仅将以下条目转换为：“ rpc_params”：[value] 或“ rpc_params”：[[value]] JSON还是字符串？问题在于，与日志完全相同的rpc_params与JSON一起出现在日志中，但在其他情况下则作为数组出现。

使用文件拍将日志发送到elasticsearch，它们首先通过logstash。我的logstash过滤器如下所示：

filter {
  if "myapp" in [tags] {
    json {
      source => "message"
    }
    }
  }
}

更多，日志中还有一些看起来像这样的条目：“ rpc_params”：[[12312312，12313123]]

基本上，我认为最好的方法是创建一个过滤器，将 [] 和 [[]] 转换为”

我尝试了使用mutate过滤器，甚至gsub过滤器，一些ruby编解码器，但似乎没有任何效果，或者可能是我做错了。

任何帮助将不胜感激。

Logstash过滤器将python Cellery日志解析为json

0 个答案: