Amazon Cloudwatch使用JSON字段记录见解

Question

我试图将Logs Insights与其中一个字段中包含JSON的数据一起使用，并解析JSON字段

当我通过入门代码深入了解数据时，数据如下所示

fields @timestamp, @message
| sort @timestamp desc
| limit 25

如何轻松地在嵌套的JSON中提取path变量以对其进行聚合？通过查看一些文档，我认为@message.path可以工作，但事实并非如此。有谁成功解析了Insights中的JSON日志

编辑：我的数据示例

#
@timestamp
@message
1
2018-12-19 23:42:52.000
I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"user,tags,promotions,company_sector,similar_professionals.tags,similar_professionals.user","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@logStream  i-05d1d61ab853517a0
@message  I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}
@timestamp  1545262972000
2
2018-12-19 23:42:16.000
I, [2018-12-19T23:42:16.723472 #851] INFO -- : [ea712503-eb86-4a6e-ab38-ddbcd6c2b4d0] {"method":"GET","path":"/api/v1/heartbeats/new","format":"json","controller":"API::V1::Public::HeartbeatsController","action":"new","status":201,"duration":9.97,"view":3.2,"time":"2018-12-19T23:42:16.712+00:00","params":{"format":"json","compress":false},"@timestamp":"2018-12-19T23:42:16.722Z","@version":"1","message":"[201] GET /api/v1/heartbeats/new (API::V1::Public::HeartbeatsController#new)"}

Answer 1

以@pyb insights为基础，我能够使用parse @message '"path":"*"' as path从@message中任何位置提取路径。

您可以继续通过传递另一个parse @message '"method":"*"' as method来获取方法，而无需担心排序，因为这是在@message上进行的第二次全局纯文本搜索

如果您的@message是：

I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"}

使用：

parse @message '"path":"*"' as path | parse @message '"method":"*"' as method

将导致以下字段：path = '/api/v1/professionals/ID'和method = 'GET'

请注意，这仍然只是字符串解析，因此，它没有嵌套键的概念，例如params.format找不到json，但是只要使用format，只要format中的任何地方都没有另一个@message字符串。

还要注意，这是针对Insights在消息中未发现JSON的情况。我相信@pyb在this answer中是指这种情况。使用以下格式也无法找到我的日志

info - Request: {"method":"POST","path":"/auth/login/","body":{"login":{"email":"email@example.com","password":"********"}},"uuid":"36d76df2-aec4-4549-8b73-f237e8f14e23","ip":"*.*.*.*"}

Answer 2

您可以使用parse命令提取字段。

如果@message是

I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}

然后您像这样提取字段：

fields @timestamp, @message
| parse "I, [*T*] INFO -- : {"method":"*"}" as @date, @time, @method
| filter method=GET
| sort @timestamp desc
| limit 20

目前该文档还很少。我可以用正则表达式替换通配符*来获得结果，但是解析失败。

Answer 3

CloudWatch Insights日志会自动发现以下日志类型的字段：

  

Lambda日志

     

CloudWatch Logs Insights自动发现Lambda日志中的日志字段，但仅针对每个日志事件中的第一个嵌入式JSON片段（注意：重点是我的）。如果Lambda日志事件包含多个JSON片段，则可以使用parse命令解析和提取日志字段。有关更多信息，请参阅JSON日志中的字段。

     

CloudTrail日志

     

请参见fields in JSON logs。

来源： Supported Logs and Discovered Fields

如果@message是I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}

然后您可以选择和过滤字段，如下所示：

fields @timestamp, @message, method
| filter method = "GET"
| sort @timestamp desc

它也适用于嵌套字段，即params.format = "json"或results.0.firstName = "Paul"。

Answer 4

还有另一个parse，借助正则表达式

假设您的@message是：

I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/"}

您可以提取方法

fields @timestamp, @message
| parse @message /\"method\":\"(?<method_type>.*?)\"/

Answer 5

parse命令中的此正则表达式查询对您有帮助吗？

filter @message like / \"path\":\"/
| parse @message /(?<@endpt>((\/[a-zA-Z0-9_{}()-?]+){1,}))/

祝你好运！