流利的解析部分json日志

时间：2020-04-24 08:51:58

标签： json parsing fluentd

我有来自Apache Druid的以下日志

{"timeMillis":1587714600921,"thread":"qtp150208546-149","level":"INFO","loggerName":"org.apache.druid.server.log.LoggingRequestLogger","message":"2020-04-24T07:50:00.798Z\t10.2.73.23\t{\"queryType\":\"segmentMetadata\",\"dataSource\":{\"type\":\"table\",\"name\":\"tableB\"},\"intervals\":{\"type\":\"LegacySegmentSpec\",\"intervals\":[\"1999-01-01T00:00:00.000Z/2114-01-01T00:00:00.000Z\"]},\"toInclude\":{\"type\":\"all\"},\"merge\":true,\"context\":{\"queryId\":\"ed1e7129-1e3f-438d-acbb-04f11d292eb5\"},\"analysisTypes\":[\"aggregators\"],\"usingDefaultInterval\":false,\"lenientAggregatorMerge\":false,\"descending\":false,\"granularity\":{\"type\":\"all\"}}\t{\"query/time\":122,\"query/bytes\":4712,\"success\":true,\"identity\":\"allowAll\"}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger"}
{"timeMillis":1587714600952,"thread":"qtp150208546-119","level":"INFO","loggerName":"org.apache.druid.server.log.LoggingRequestLogger","message":"2020-04-24T07:50:00.941Z\t10.2.73.23\t{\"queryType\":\"segmentMetadata\",\"dataSource\":{\"type\":\"table\",\"name\":\"test\"},\"intervals\":{\"type\":\"LegacySegmentSpec\",\"intervals\":[\"1999-01-01T00:00:00.000Z/2114-01-01T00:00:00.000Z\"]},\"toInclude\":{\"type\":\"all\"},\"merge\":true,\"context\":{\"queryId\":\"6368e8c4-e18d-4a29-97cc-df2e0aadd02e\"},\"analysisTypes\":[\"aggregators\"],\"usingDefaultInterval\":false,\"lenientAggregatorMerge\":false,\"descending\":false,\"granularity\":{\"type\":\"all\"}}\t{\"query/time\":10,\"query/bytes\":4710,\"success\":true,\"identity\":\"allowAll\"}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger"}
{"timeMillis":1587714662763,"thread":"qtp150208546-131","level":"INFO","loggerName":"org.apache.druid.server.log.LoggingRequestLogger","message":"2020-04-24T07:51:02.694Z\t\t{\"queryType\":\"topN\",\"dataSource\":{\"type\":\"table\",\"name\":\"tableA\"},\"virtualColumns\":[],\"dimension\":{\"type\":\"default\",\"dimension\":\"key\",\"outputName\":\"d0\",\"outputType\":\"STRING\"},\"metric\":{\"type\":\"numeric\",\"metric\":\"a0\"},\"threshold\":100,\"intervals\":{\"type\":\"intervals\",\"intervals\":[\"2020-03-05T07:51:02.000Z/146140482-04-24T15:36:27.903Z\"]},\"filter\":null,\"granularity\":{\"type\":\"all\"},\"aggregations\":[{\"type\":\"count\",\"name\":\"a0\"}],\"postAggregations\":[],\"context\":{\"queryId\":\"2084372b-f9ec-43c4-a5c7-bb6c28a738fc\",\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\"},\"descending\":false}\t{\"query/time\":68,\"query/bytes\":-1,\"success\":true,\"identity\":\"allowAll\"}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger"}
{"timeMillis":1587714662763,"thread":"qtp150208546-131","level":"INFO","loggerName":"org.apache.druid.server.log.LoggingRequestLogger","message":"2020-04-24T07:51:02.642Z\t10.2.64.24\t\t{\"sqlQuery/time\":121,\"sqlQuery/bytes\":2095,\"success\":true,\"context\":{\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\",\"nativeQueryIds\":\"[2084372b-f9ec-43c4-a5c7-bb6c28a738fc]\"},\"identity\":\"allowAll\"}\t{\"query\":\"SELECT * FROM (SELECT\\n  \\\"key\\\",\\n  COUNT(*) AS \\\"Count\\\"\\nFROM \\\"tableA\\\"\\nWHERE \\\"__time\\\" >= CURRENT_TIMESTAMP - INTERVAL '50' DAY\\nGROUP BY 1\\nORDER BY \\\"Count\\\" DESC\\n) LIMIT 100\",\"context\":{\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\",\"nativeQueryIds\":\"[2084372b-f9ec-43c4-a5c7-bb6c28a738fc]\"}}","endOfBatch":false,"loggerFqcn":"org.apache.logging.slf4j.Log4jLogger"}

我想使用JSON解析器解析键message，以获取每个提取的字段和值

"2020-04-24T07:50:00.798Z\t10.2.73.23\t{\"queryType\":\"segmentMetadata\",\"dataSource\":{\"type\":\"table\",\"name\":\"tableA\"},\"intervals\":{\"type\":\"LegacySegmentSpec\",\"intervals\":[\"1999-01-01T00:00:00.000Z/2114-01-01T00:00:00.000Z\"]},\"toInclude\":{\"type\":\"all\"},\"merge\":true,\"context\":{\"queryId\":\"ed1e7129-1e3f-438d-acbb-04f11d292eb5\"},\"analysisTypes\":[\"aggregators\"],\"usingDefaultInterval\":false,\"lenientAggregatorMerge\":false,\"descending\":false,\"granularity\":{\"type\":\"all\"}}\t{\"query/time\":122,\"query/bytes\":4712,\"success\":true,\"identity\":\"allowAll\"}"
"2020-04-24T07:50:00.941Z\t10.2.73.23\t{\"queryType\":\"segmentMetadata\",\"dataSource\":{\"type\":\"table\",\"name\":\"test\"},\"intervals\":{\"type\":\"LegacySegmentSpec\",\"intervals\":[\"1999-01-01T00:00:00.000Z/2114-01-01T00:00:00.000Z\"]},\"toInclude\":{\"type\":\"all\"},\"merge\":true,\"context\":{\"queryId\":\"6368e8c4-e18d-4a29-97cc-df2e0aadd02e\"},\"analysisTypes\":[\"aggregators\"],\"usingDefaultInterval\":false,\"lenientAggregatorMerge\":false,\"descending\":false,\"granularity\":{\"type\":\"all\"}}\t{\"query/time\":10,\"query/bytes\":4710,\"success\":true,\"identity\":\"allowAll\"}"
"2020-04-24T07:51:02.694Z\t\t{\"queryType\":\"topN\",\"dataSource\":{\"type\":\"table\",\"name\":\"tableA\"},\"virtualColumns\":[],\"dimension\":{\"type\":\"default\",\"dimension\":\"key\",\"outputName\":\"d0\",\"outputType\":\"STRING\"},\"metric\":{\"type\":\"numeric\",\"metric\":\"a0\"},\"threshold\":100,\"intervals\":{\"type\":\"intervals\",\"intervals\":[\"2020-03-05T07:51:02.000Z/146140482-04-24T15:36:27.903Z\"]},\"filter\":null,\"granularity\":{\"type\":\"all\"},\"aggregations\":[{\"type\":\"count\",\"name\":\"a0\"}],\"postAggregations\":[],\"context\":{\"queryId\":\"2084372b-f9ec-43c4-a5c7-bb6c28a738fc\",\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\"},\"descending\":false}\t{\"query/time\":68,\"query/bytes\":-1,\"success\":true,\"identity\":\"allowAll\"}"
"2020-04-24T07:51:02.642Z\t10.2.64.24\t\t{\"sqlQuery/time\":121,\"sqlQuery/bytes\":2095,\"success\":true,\"context\":{\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\",\"nativeQueryIds\":\"[2084372b-f9ec-43c4-a5c7-bb6c28a738fc]\"},\"identity\":\"allowAll\"}\t{\"query\":\"SELECT * FROM (SELECT\\n  \\\"key\\\",\\n  COUNT(*) AS \\\"Count\\\"\\nFROM \\\"tableA\\\"\\nWHERE \\\"__time\\\" >= CURRENT_TIMESTAMP - INTERVAL '50' DAY\\nGROUP BY 1\\nORDER BY \\\"Count\\\" DESC\\n) LIMIT 100\",\"context\":{\"sqlQueryId\":\"e137d0ef-38ba-4af5-ac0e-5cdf86439ace\",\"nativeQueryIds\":\"[2084372b-f9ec-43c4-a5c7-bb6c28a738fc]\"}}"
您可以看到

开头不是JSON格式。但是我只想提取JSON部分。
流利的版本：v1.8.1
我正在尝试这种Fluentd配置：

<filter kubernetes.var.log.containers.druid-brokers-**.log>
      @type parser
      key_name $["log"]["message"]
      reserve_data true
      remove_key_name_field true
      hash_value_field parsed
      <parse>
        @type json
      </parse>
</filter>

但是我无法理解以下错误消息：

[warn]: #0 dump an error event: error_class=TypeError error="String does not have #dig method"

感谢您的帮助。

关于， 文森特

1 个答案:

答案 0 :(得分：0)

我认为 message 字段必须访问为：

key_name $["message"]

我也遇到了同样的错误，结果我尝试访问的日志记录中的元素不正确，即路径不存在。

相关问题
最新问题