Question

我想在Kibana4的瓷砖地图中显示地理位置字段。使用标准/自动logstash geoip映射到elasticsearch，一切正常。

但是，在创建非标准geoip字段时，我不太确定如何在logstash中自定义elasticsearch-template.json以便在elasticsearch中正确表示此字段，以便可以在Kibana4中选择它以创建tile图

当然，自定义标准模板不是最好的方法 - 更好地创建自定义模板并在logstash.conf的elasticsearch输出中指向它。我很快就想检查如何定义映射，所以我修改了标准模板。

我的logstash.conf：

input {
    tcp {
        port => 514
        type => syslog
    }
    udp {
        port => 514
        type => syslog
    }
}

filter {
    # Standard geoip field is automatically mapped by logstash to 
    # elastic search by using the elasticsearch-template.json file
    geoip { source => "host" }

    grok {
        match => [ 
            "message", "<%{POSINT:syslog_pri}>%{YEAR} %{SYSLOGTIMESTAMP:syslog_timestamp} %{DATA:device} <%{POSINT:status}> %{WORD:activity} %{DATA:inout} \(%{DATA:msg}\) Src:%{IPV4:src} SPort:%{INT:sport} Dst:%{IPV4:dst} DPort:%{INT:dport} IPP:%{INT:ipp} Rule:%{INT:rule} Interface:%{WORD:iface}",
            "message", "<%{POSINT:syslog_pri}>%{YEAR} %{SYSLOGTIMESTAMP:syslog_timestamp} %{DATA:device} <%{POSINT:status}> %{WORD:activity} %{DATA:inout} \(%{DATA:msg}\) Src:%{IPV4:src} Dst:%{IPV4:dst} IPP:%{INT:ipp} Rule:%{INT:rule} Interface:%{WORD:iface}",                    
            "message", "<%{POSINT:syslog_pri}>%{YEAR} %{SYSLOGTIMESTAMP:syslog_timestamp} %{DATA:device} <%{POSINT:status}> %{WORD:activity} %{DATA:inout} \(%{DATA:msg}\) Src:%{IPV4:src} Dst:%{IPV4:dst} Type:%{POSINT:type} Code:%{INT:code} IPP:%{INT:ipp} Rule:%{INT:rule} Interface:%{WORD:iface}"
        ]
    }
    # Is not mapped automatically by logstash in that it can be 
    # chosen in Kibana4 for tile map creation
    geoip {
        source => "src"
        target => "src_geoip"
    }   
}

output {
    elasticsearch {
        host => "localhost"
        protocol => "http"
    }
}

我的... logstash-1.4.2 \ lib \ logstash \ outputs \ elasticsearch \ elasticsearch-template.json：

    {
  "template" : "logstash-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true},
       "dynamic_templates" : [ {
         "string_fields" : {
           "match" : "*",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true,
               "fields" : {
                 "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
               }
           }
         }
       } ],
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" },
         "geoip"  : {
           "type" : "object",
             "dynamic": true,
             "path": "full",
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         },
         "src_geoip"  : {
           "type" : "object",
             "dynamic": true,
             "path": "full",
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         }
       }
    }
  }
}

更新：我还没知道这个json文件何时应用于elasticsearch。我按照this question中列出的提示进行操作，并将json文件复制到elasticsearch目录中的config / templates文件夹中。删除indizes并重新启动elasticsearch后，模板已成功应用。

无论如何，字段＆＃34; src_geoip.location＆＃34;仍然没有出现在Kibana4的瓷砖地图创建表单中（只有标准的geoip.location字段）。

Answer 1

编辑模板后尝试覆盖模板。配置更改后，在Kibana中重新创建索引。

output {
   elasticsearch {
       template_overwrite => "true"
        ...
   }
}

Answer 2

您还需要在elasticsearch实例的索引模板中为src_geoip对象添加对象。要为匹配＆＃34; logstash-netflow - *＆＃34;的所有索引设置默认模板，请在elasticsearch实例上执行以下操作：

curl -XPUT localhost:9200/_template/logstash-netflow -d '{
"template" : "logstash-netflow-*",
"mappings" : {
  "_default_" : {
    "_all" : {
      "enabled" : false
    },
    "properties" : {
      "@timestamp" : { "index" : "analyzed", "type" : "date" },
      "@version" : { "index" : "analyzed", "type" : "integer" },
      "src_geoip" : {
        "dynamic" : true,
        "type" : "object",
        "properties" : {
          "area_code" : { "type" : "long" },
          "city_name" : { "type" : "string" }, 
          "continent_code" : { "type" : "string" }, 
          "country_code2" : { "type" : "string" },
          "country_code3" : { "type" : "string" },
          "country_name" : { "type" : "string" },
          "dma_code" : { "type" : "long" },
          "ip" : { "type" : "string" },
          "latitude" : { "type" : "double" },
          "location" : { "type" : "double" },
          "longitude" : { "type" : "double" },
          "postal_code" : { "type" : "string" },
          "real_region_name" : { "type" : "string" },
          "region_name" : { "type" : "string" },
          "timezone" : { "type" : "string" }
        }
      },
      "netflow" : { ....snipped......
      }
    }
  }
}}'

如何在logstash中使用elasticsearch映射geoip字段，以便在Kibana4的tile map中显示它

2 个答案: