Question

我收到此错误：

无法将事件索引到Elasticsearch。 {：状态=> 400，：action => [“ index”，{：_id => nil，：_index =>“ nginx-access-2018-06-15”，：_type =>“ doc”，：_routing => nil}，＃]，：response => {“ index” => {“ _ index” =>“ nginx-access-2018-06-15”， “ _type” =>“ doc”，“ _id” =>“ jo-rfGQBDK_ao1ZhmI8B”，“状态” => 400， “ error” => {“ type” =>“ illegal_argument_exception”， “ reason” =>“ [geoip.location]被定义为映射中的对象[doc] 但是此名称已用于其他类型的字段”}}}}

我遇到上述错误，但不明白为什么，这正在加载到没有数据的全新ES实例中。这是插入的第一条记录。为什么会出现此错误？这是配置：

input {

  file {
    type => "nginx-access"
    start_position => "beginning"
    path => [ "/var/log/nginx-archived/access.log.small"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  if [type] == "nginx-access" {
       grok {
         patterns_dir => "/etc/logstash/patterns"
         match => { "message" => "%{NGINX_ACCESS}" }
         remove_tag => ["_grokparsefailure"]
       }
       geoip {
         source => "visitor_ip"
       }
       date {
         # 11/Jun/2018:06:23:45 +0000
         match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
         target => "@request_time"
       }
       if "_grokparsefailure" not in [tags] {
            ruby {
              code => "
                thetime = event.get('@request_time').time
                event.set('index_date', 'nginx-access-' + thetime.strftime('%Y-%m-%d'))
              "
            }
          }
         if "_grokparsefailure" in [tags] {
              ruby {
                code => "
                  event.set('index_date', 'nginx-access-error')
                "
              }
            }
     }

}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "%{index_date}"
    template => "/etc/logstash/templates/nginx-access.json"
    template_overwrite => true
    manage_template => true
    template_name => "nginx-access"
  }
  stdout { }
}

以下是示例记录：

{
            "method" => "GET",
          "@version" => "1",
             "geoip" => {
         "continent_code" => "AS",
               "latitude" => 39.9289,
           "country_name" => "China",
                     "ip" => "220.181.108.103",
               "location" => {
             "lon" => 116.3883,
             "lat" => 39.9289
         },
            "region_code" => "11",
            "region_name" => "Beijing",
              "longitude" => 116.3883,
               "timezone" => "Asia/Shanghai",
              "city_name" => "Beijing",
          "country_code2" => "CN",
          "country_code3" => "CN"
     },
        "index_date" => "nginx-access-2018-06-15",
            "ignore" => "\"-\"",
             "bytes" => "2723",
           "request" => "/wp-login.php",
     "@request_time" => 2018-06-15T06:29:40.000Z,
           "message" => "220.181.108.103 - - [15/Jun/2018:06:29:40 +0000] \"GET /wp-login.php HTTP/1.1\" 200 2723 \"-\" \"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)\"",
              "path" => "/var/log/nginx-archived/access.log.small",
        "@timestamp" => 2018-07-09T01:32:56.952Z,
              "host" => "ab1526efddec",
        "visitor_ip" => "220.181.108.103",
         "timestamp" => "15/Jun/2018:06:29:40 +0000",
          "response" => "200",
          "referrer" => "\"Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)\"",
       "httpversion" => "1.1",
              "type" => "nginx-access"
 }

Answer 1

根据以下答案找出答案：

https://www.elastic.co/guide/en/elasticsearch/reference/6.x/removal-of-types.html#_schedule_for_removal_of_mapping_types

基本问题是，对于每个Elasticsearch索引，即使记录是不同的类型，每个字段也必须是同一类型。

也就是说，如果我有一个人{ "status": "A" }作为文本存储，那么我无法在同一索引中将一个汽车{ "status": 23 }的记录存储为数字。根据上面链接中的信息，我为每个索引存储一种“类型”。

我的Logstash输出部分如下所示：

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "%{index_date}"
    # Can test loading this with:
    # curl -XPUT -H 'Content-Type: application/json' -d@/docker-elk/logstash/templates/nginx-access.json http://localhost:9200/_template/nginx-access
    template => "/etc/logstash/templates/nginx-access.json"
    template_overwrite => true
    manage_template => true
    template_name => "nginx-access"
  }
  stdout { }
}

我的模板如下：

{
  "index_patterns": ["nginx-access*"],
  "settings": {

  },
  "mappings": {
    "doc": {
      "_source": {
        "enabled": true
      },
      "properties": {
    "type" : { "type": "keyword" },
    "response_time": { "type": "float" },
        "geoip" : {
      "properties" : {
        "location": {
              "type": "geo_point"
            }
          }
    }
      }
    }
  }
}

我还在上面链接中描述的每个索引模式使用一种类型。

geoip.location被定义为映射[doc]中的对象，但此名称已用于其他类型的字段

1 个答案: