Question

当前日志类型如下：

{"event": "Task succeeded", "id": "13e495e3-1a0a-4612-bd11-34293ada3133", "level": "info", "logger": "celery.worker", "name": "kraken.tasks.RPCTask", "rpc_endpoint": "routing", "rpc_method": "route_sms", "rpc_params": [32425657567, 606104377393580], "runtime": 0.48229960491880774, "service": "core", "timestamp": "2018-10-24T05:30:21.557987Z"}
{"event": "Received task", "id": "13e495e3-1a0a-4612-bd11-34293ada3133", "level": "info", "logger": "celery.worker", "name": "kraken.tasks.RPCTask", "rpc_endpoint": "routing", "rpc_method": "route_sms", "rpc_params": [657547567567, 606104377393580], "timestamp": "2018-10-24T05:30:21.074257Z"}
{"event": "Task accepted", "id": "4a0e3e50-3876-4850-a637-889ea36dedd3", "level": "info", "logger": "celery.worker", "name": "kraken.tasks.SilentRPCTask", "pid": 140530763197808, "rpc_endpoint": "analytics.fact", "rpc_method": "add", "rpc_params": {"city_id": null, "correlation_id": "08ee597c-8ee4-48f2-87b7-89ef4c844069", "country_id": null, "dst_is_platform": true, "dst_ln": 56546456, "dst_mid": null, "dst_operator_id": null, "dst_package_id": null, "dst_user_id": null, "duration": null, "ignore_errors": true, "medium": null, "platform_event": "sms-in", "service": "wallet", "service_event": null, "service_path": null, "sms_in_size": 1, "sms_out_size": null, "src_is_platform": false, "src_ln": 4353434543, "src_mid": null, "src_operator_id": 35345, "src_package_id": null, "src_user_id": 30639, "ts_end": null, "ts_start": null}, "timestamp": "2018-10-24T05:30:18.675206Z"}

您可以看到，除某些条目外，它为JSON格式。这里的问题是，我大多数时候都使用JSON格式的rpc_params，但是当rpc_params看起来像这样时有一些条目：“ rpc_params”：[[]] 或“ rpc_params”： [] 或“ rpc_params”：[[123123123,12312312,123123123]]

我应该在logstash中使用哪个过滤器？日志与filebeat一起传送到logstash，然后弹性搜索到特定索引。我当前的logstash过滤器如下所示：

filter {
    if "celleryapp" in [tags] {

        mutate {
            gsub => ["rpc_params", "[\[\]]", ""]
            gsub => ["rpc_params", "[\[\[\]\]]", ""]
        }

        mutate {
            split => { "rpc_params" => "," }
        }

        json {
                source => "message"
        }
    }
}

无论是否使用mutate过滤器，我总是在logstash中收到此错误：

[2018-10-24T05:34:57,878][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"celleryapp-2018.10.24", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x47ab2458>], :response=>{"index"=>{"_index"=>"celleryapp-2018.10.24", "_type"=>"doc", "_id"=>"XfuRpGYBiD1kYJIQm9JM", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [rpc_params] tried to parse field [null] as object, but found a concrete value"}}}}
[2018-10-24T05:34:57,879][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"celleryapp-2018.10.24", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x44883a69>], :response=>{"index"=>{"_index"=>"celleryapp-2018.10.24", "_type"=>"doc", "_id"=>"XvuRpGYBiD1kYJIQm9JM", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [rpc_params] tried to parse field [null] as object, but found a concrete value"}}}}

问题在于，完全相同的 rpc_params 是 JSON 格式的一个，然后是数组。

我使用mutate过滤器尝试删除[] [[]]。我可能需要仅当 rpc_params不是JSON格式

时才需要对rpc_params应用的过滤器

Logstash从数组创建json

0 个答案: