Question

我使用logstash将数据插入到弹性搜索中，并且conf文件看起来像

  grok {
             match => [
         "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}",
         "message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra_fields}"
         ]
         overwrite => [ "message" ]
      }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  geoip {
      source => "clientip"
    }

现在，Elastic Search映射已完成

{                                                                                                                                                        
   "response" => "200",                                                                                                                              
      "geoip" => {                                                                                                                                   
          "timezone" => "Asia/Kolkata",                                                                                                              
     "country_code3" => "IN",                                                                                                                        
          "location" => {                                                                                                                            
        "lon" => 80.2833,                                                                                                                            
        "lat" => 13.0833                                                                                                                             
    },                                                                                                                                               
       "region_code" => "TN",                                                                                                                        
      "country_name" => "India",                                                                                                                     
         "longitude" => 80.2833,                                                                                                                     
         "city_name" => "Chennai",                                                                                                                   
       "region_name" => "Tamil Nadu",                                                                                                                
          "latitude" => 13.0833,                                                                                                                     
    "continent_code" => "AS",                                                                                                                        
       "postal_code" => "600073",                                                                                                                    
     "country_code2" => "IN",                                                                                                                        
                "ip" => "122.15.151.189"                                                                                                             
},                                                                                                                                                   
  "timestamp" => "31/May/2019:05:12:22 -0700",                                                                                                       
    "request" => "/favicon.ico",                                                                                                                     
      "ident" => "-",                                                                                                                                
       "auth" => "-",                                                                                                                                
      "agent" => "\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36\"",          
   "referrer" => "\"http://example.com/\"",            
 "@timestamp" => 2019-05-31T12:12:22.000Z,                                                                                                           
"httpversion" => "1.1",                                                                                                                              
       "verb" => "GET"

}

现在，当尝试在KIBANA中加载可视化程序时，geoHash无法正常运行，因为geoip.location未定义为“ type”：“ geo_point”。我已经插入了整个数据。因此，我可以做些什么来修改映射并更新索引。我了解的是，我需要在grok中添加一个额外的字段，并将其设置为“ geoip.location”字段，然后再次重新解析整个日志。但是有什么方法可以在现有的弹性索引中创建一列并从2列合并数据？

Like geoip.location2 merged from geoip.location.lon and location.lat

Answer 1

通常，您无法修改索引中已经存在的字段的映射
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#_updating_existing_field_mappings

但是，您可以通过添加另一个字段来更新索引的映射。您可以检查exact syntax based on the version of Elastic that you're using，但是对我来说，在Elastic 6.2上，以下工作有效：

PUT geo_test1/_mapping/geo_test1
{
  "properties": {
    "geoip": {
      "properties": {
        "location_geo": {
          "type": "geo_point"
        }
      }
    }
  }
}

然后，您可以使用update_by_query和一个小的脚本来根据geoip.location中已经存在的数据在新的“ location_geo”字段中填充数据：

POST geo_test1/_update_by_query
{
  "script": {
    "source": "String lat = ctx._source.geoip.location.lat.toString(); String lon = ctx._source.geoip.location.lon.toString(); ctx._source.geoip.location_geo = lat + ',' + lon",
    "lang": "painless"
  },
  "query": {
    "match_all": {}
  }
}

apachelog的logstash解析后，geipip.location丢失，Kibana Map Visualizer无法正常工作

1 个答案: