Question

如何更新logstash或elasticsearch模板，这样可以避免在logstash服务器上显示此mapper_parsing_exception错误。试图麋鹿进行awscloudtrail日志记录，但似乎每一步都遇到障碍。

我也了解用grok完成的操作，但不知道如何做！

[2018-10-06T00:28:19,948][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.10.06", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x73d7cfa2>], :response=>{"index"=>{"_index"=>"cloudtrail-2018.10.06", "_type"=>"doc", "_id"=>"pIPGRmYBZjQ3shBzZDlQ", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [Records.requestParameters.ebsOptimized] tried to parse field [ebsOptimized] as object, but found a concrete value"}}}}

第二次错误

2018-10-06T00:28:33,047][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.10.06", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x7c5a7518>], :response=>{"index"=>{"_index"=>"cloudtrail-2018.10.06", "_type"=>"doc", "_id"=>"JIPGRmYBZjQ3shBzmTqd", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [Records.apiVersion]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2017_03_25\" is malformed at \"_03_25\

这是我的模板和默认动态映射

{
  "cloudtrail": {
    "order": 0,
    "version": 60001,
    "index_patterns": [
      "cloudtrail-*"
    ],
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "6000"
          }
        },
        "refresh_interval": "5s"
      }
    },
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "match_mapping_type": "string",
              "mapping": {
                "type": "text",
                "norms": false
              }
            }
          },
          {
            "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "keyword"
          },
          "geoip": {
            "dynamic": true,
            "properties": {
              "ip": {
                "type": "ip"
              },
              "location": {
                "type": "geo_point"
              },
              "latitude": {
                "type": "half_float"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          }
        }
      }
    },
    "aliases": {}
  }
}

映射，不确定为什么这么大

"apiVersion": {
                "type": "date"
              },
              "awsRegion": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }

更新1 ：

@Amir masud zarebidaki，谢谢。我认为elasticsearch也会动态地进行所有映射并进行优化，这对于Elasticsearch概念还是很新的。但是，通过在过滤器中添加以下内容，我确实摆脱了他的日期格式错误

mutate {
             gsub => [              "eventSource", "\.amazonaws\.com$", "",
              "apiVersion", "_", "-"        ]

，但不确定是否正确，因为现在丢失了一些数据并且我在日志中以

的形式出现了新错误

[2018-10-08T13:26:16,000][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.10.08", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x30ef00f4>], :response=>{"index"=>{"_index"=>"cloudtrail-2018.10.08", "_type"=>"doc", "_id"=>"aknbU2YBeGi09oGfWDua", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [responseElements.role]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:230"}}}}}

cloudtrail logstash mapper_parsing_exception状态为400

0 个答案: