Question

我得到了一些日志：

2015-09-25 12:07:55.441 INFO 17328 --- [][][][] 
XXX.YYY.SomeClass : Someone request in CityX!

我将其导入Elasticsearch：

{
    "_index": "logstash-2015.09.25",
    "_type": "redis-input",
    "_id": "AVADGRo7JaVbcBhehzEj",
    "_score": 1,
    "_source": {
        "@timestamp": "2015-09-25T12:21:24.616+08:00",
        "@version": 1,
        "message": "Someone request in CityX!",
        "logger_name": "XXX.YYY.SomeClass",
        "thread_name": "pool-22-thread-1",
        "level": "INFO",
        "level_value": 20000,
        "HOSTNAME": "host",
        "host": "192.168.5.194: 57154",
        "type": "redis-input"
    }
}

我只想将CityX（我的日志中发生的任何城市，并假设我们可以获得每个城市的纬度和经度）映射到GeoPoint中的Elasticsearch，以便我们可以通过Kibana在地图中显示用户请求的计数。我该怎么做？

整个管道：

logstash(:4560) --> redis(:6379) --> logstash-indexer --> elasticsearch (:9200)

配置：

Logstash - ＆gt; Redis的：

input {
  tcp {
    port => 4560
    codec => json_lines
  }
}

output {

  redis {
    host => "10.0.40.155"
    port => 6379
    data_type => "list"
    key => "key_count"
  }
}

Redis - ＆gt; Logstash - ＆gt; Elasticsearch：

input {
  redis {
    host => "127.0.0.1"
    port => 6379
    type => "redis-input"
    data_type => "list"
    key => "key_count"
  }
}

output {
  stdout {}
  elasticsearch {
    host => "10.0.40.156"
    cluster => "elasticsearch"
    codec => "json"
    protocol => "http"
  }
}

Answer 1

我的Java程序记录City,Longitude,Latitude：

我的日志示例：

ChinaUnicom Zhejiang Hangzhou 30.29294,120.10956 REQUEST
ChinaUnicom Zhejiang Hangzhou 30.29294,120.10956 REQUEST
ChinaTelecom Zhejiang Hangzhou 30.29294,120.10956 REQUEST

Zhejiang是中国的一个省，Hangzhou是Zhejiang的城市。

我首先添加grok过滤器来解析日志，然后使用add_field将其转换为Kibana可以识别的geo_point。

input {
  redis {
    host => "127.0.0.1"
    port => 6379
    type => "redis-input"
    data_type => "list"
    key => "key_count"
  }
}

filter {
  grok {
    match => { "message" => "%{WORD:carrier} %{WORD:province} %{WORD:city} %{BASE10NUM:latitude},%{BASE10NUM:longitude} %{WORD:geo_message}"}
    add_field => {"geoip.location" => "%{latitude},%{longitude}"}
  }
}

output {
  stdout {}
  elasticsearch {
    host => "10.0.40.156"
    cluster => "elasticsearch"
    codec => "json"
    protocol => "http"
  }
}

将城市名称映射到从Logstash到Elasticsearch的GeoPoint

1 个答案: