Question

当我在Kibana中看到结果时，我发现JSON中没有字段，而且message字段仅包含"status" : "FAILED"。

是否可以从json解析字段并在Kibana中显示它们？我有以下配置：

input {
  file {
    type => "json"
    path => "/home/logstash/test.json"
    codec => json
    sincedb_path => "/home/logstash/sincedb"
  }
} 

output {
  stdout {}
  elasticsearch {
    protocol => "http"
    codec => "json"
    host => "elasticsearch.dev"
    port => "9200"
  }
}

关注JSON文件：

[{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182,"stop":1419621640491,"duration":17309},"severity":"NORMAL","status":"FAILED"},{"uid":"a88c89b377aca0c9","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182,"stop":1419621640634,"duration":17452},"severity":"NORMAL","status":"FAILED"},{"uid":"32c3f8b52386c85c","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623185,"stop":1419621640826,"duration":17641},"severity":"NORMAL","status":"FAILED"}]

Answer 1

是。你需要在你的配置中添加一个过滤器，就像这样。

filter{
    json{
        source => "message"
    }
}

它在文档here

中描述得非常好

修改 json编解码器似乎不喜欢传入数组。单个元素可以使用此配置：

<强>输入：

{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182, "stop":1419621640491,"duration":17309 }, "severity":"NORMAL", "status":"FAILED" }

Logstash结果：

{ "message" => "{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }", "@version" => "1", "@timestamp" => "2015-02-26T23:25:12.011Z", "host" => "emmet.local", "uid" => "441d1d1dd296fe60", "name" => "test_buylinks", "title" => "Testbuylinks", "time" => { "start" => 1419621623182, "stop" => 1419621640491, "duration" => 17309 }, "severity" => "NORMAL", "status" => "FAILED"

}

现在有一个数组：

<强>输入

[{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182, "stop":1419621640491,"duration":17309 }, "severity":"NORMAL", "status":"FAILED" }, {"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182, "stop":1419621640491,"duration":17309 }, "severity":"NORMAL", "status":"FAILED" }]

<强>结果：

Trouble parsing json {:source=>"message", :raw=>"[{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }, {\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }]", :exception=>#<TypeError: can't convert Array into Hash>, :level=>:warn} { "message" => "[{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }, {\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182, \"stop\":1419621640491,\"duration\":17309 }, \"severity\":\"NORMAL\", \"status\":\"FAILED\" }]", "@version" => "1", "@timestamp" => "2015-02-26T23:28:21.195Z", "host" => "emmet.local", "tags" => [ [0] "_jsonparsefailure" ] }

这看起来像编解码器中的错误，您可以将消息更改为对象而不是数组吗？

Answer 2

尝试使用json_lines编解码器代替json。这必须是最近添加的。在您的特定情况下，您首先要将输出从json列表更改为换行符分隔的json。

http://logstash.net/docs/1.4.0/codecs/json_lines

此编解码器将解码以换行符分隔的流式JSON。对于例如，在redis输入中解码JSON有效负载，使用json 编解码器。编码将发出以a结尾的单个JSON字符串 '\ n'

Answer 3

读取包含JSON数组的文件比应该更难。以下是工作流水线配置

input {
  exec {
    command => "cat /path/file_containing_json_array.txt"
    codec => "json"
    interval => 3600
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Logstash不解析json

3 个答案: