Question

Logstash 6.6.0

我的应用程序通过使用NewtonSoft将Dictionary序列化为json来生成日志.Json然后通过udp将其发送到logstash。

我一直在logstash中看到这些错误：

[2019-02-21T05:10:59,679][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"{\"appName\":\"redacted.Service\",\"level\":\"ERROR\",\"messageObject\":\"Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received from 711042057 with sequence number 63453 at 03:31:02.00\",\"renderedMessage\":\"Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received from with sequence number 63453 at 03:31:02.00\",\"timestampUtc\":\"2019-02-21T15:10:58.8914689Z\",\"logger\":\"redactedService\",\"thread\":\"Listening Thread 13\",\"exceptionObject\":{\"ClassName\":\"System.Exception\",\"Message\":\"Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received from 711042057 with sequence number 63453 at 03:31:02.00\",\"Data\":null,\"InnerException\":null,\"HelpURL\":null,\"StackTraceString\":null,\"RemoteStackTraceString\":null,\"RemoteStackIndex\":0,\"ExceptionMethod\":null,\"HResult\":-2146233088,\"Source\":null,\"WatsonBuckets\":null},\"exceptionObjectString\":\"System.Exception: Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received from with sequence number 63453 at 03:31:02.00\",\"userName\":\"redacted\\\\redacteds-redactedqa$\",\"domain\":\"redacted.service.exe\",\"identity\":\"\",\"pid\":14872,\"machineName\":\"redactedQA\",\"workingSet\":439525376,\"osVersion\":\"Microsoft Windows NT 6.2.9200.0\",\"properties\":{\"ServiceId\":\"1\",\"log4net:UserName\":\"redacted\\\\redacteds-redactedqa$\",\"MasterId\":\"1\",\"log4net:Identity\":\"\",\"log4net:HostName\":\"redactedQA\"}}\nSystem.Exception: Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received from 711042057 with sequence number 63453 at 03:31:02.00\r\n", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'System': was expecting ('true', 'false' or 'null')
 at [Source: (byte[])"{"appName":"redacted.Service","level":"ERROR","messageObject":"Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received from with sequence number 63453 at 03:31:02.00","renderedMessage":"Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received"[truncated 1538 bytes]; line: 2, column: 8]>}

配置：

input {
    udp {
        type => "json"
        port => 9996
    }
}

filter{
    if [type] == "json" {

        json {
            source => "message"
        }
        mutate {
            remove_field => [ "messageObject.InnerException._connectionId.ServerId._endPoint"  ]
        }
    }
}

output {
        elasticsearch{
            redacted
        }

}

示例消息：

{
"appName": "redacted.Service",
"level": "ERROR",
"messageObject": "Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received from 711042057 with sequence number 63453 at 03:31:02.00",
"renderedMessage": "Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received from with sequence number 63453 at 03:31:02.00",
"timestampUtc": "2019-02-21T15:10:58.8914689Z",
"logger": "redactedService",
"thread": "Listening Thread 13",
"exceptionObject": {
    "ClassName": "System.Exception",
    "Message": "Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received from 711042057 with sequence number 63453 at 03:31:02.00",
    "Data": null,
    "InnerException": null,
    "HelpURL": null,
    "StackTraceString": null,
    "RemoteStackTraceString": null,
    "RemoteStackIndex": 0,
    "ExceptionMethod": null,
    "HResult": -2146233088,
    "Source": null,
    "WatsonBuckets": null
},
"exceptionObjectString": "System.Exception: Error: at 2/21/2019 10:10 AM while processing customer 1479 mobile esn event datetime 2/21/2019 3:31 AM caused the following user error message Duplicate message received from with sequence number 63453 at 03:31:02.00",
"userName": "redacted\redacteds-redactedqa$",
"domain": "redacted.service.exe",
"identity": "",
"pid": 14872,
"machineName": "redactedQA",
"workingSet": 439525376,
"osVersion": "Microsoft Windows NT 6.2.9200.0",
"properties": {
    "ServiceId": "1",
    "log4net:UserName": "redacted\redacteds-redactedqa$",
    "MasterId": "1",
    "log4net:Identity": "",
    "log4net:HostName": "redactedQA"
}
}

考虑到现有的下载量最大的json包，我很难相信Newtonsoft正在产生无效的json。

如何获取可靠的logstash来解析JSON日志？

为什么LogStash无法解析好的Json

0 个答案: