我正在尝试将routingConfig升级为logstash 1.5.3。
logstash 2.3.4的logstash配置如下所示： -
1.5.3我为input {
kafka {
zk_connect => "kafka:2181"
group_id => "logstash"
topic_id => "logstash_logs2"
reset_beginning => false
consumer_threads => 3
}
}
filter {
if [app] == "walle_slowquery" or [app] == "walle_slowindex" {
ruby {
code => "event['timestamp'] = event['@timestamp']"
}
}
grok {
match => [
"timestamp", "^(?<app_log_time>%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
]
}
mutate {
rename => {
"app_log_time" => "[@metadata][app_log_time]"
}
}
}
output {
if [env] == "prod" or [env] == "common" {
elasticsearch {
index => "jabong-%{env}-%{app}-%{iver}-%{[@metadata][app_log_time]}"
cluster => "elasticsearch"
host => ["172.16.84.230:9300"]
protocol => "transport"
}
file {
path => "/var/log/shop/%{env}/%{app}/%{app}_%{host}_%{[@metadata][app_log_time]}.log"
}
stdout { codec => rubydebug }
}
}
修改了logstash配置，如下所示： -
2.3.4在elasticsearch中，我使用索引input {
kafka {
zk_connect => "kafka:2181"
group_id => "logstash"
topic_id => "logstash_logs2"
reset_beginning => false
consumer_threads => 3
}
}
filter {
if [app] == "walle_slowquery" or [app] == "walle_slowindex" {
ruby {
code => "event['timestamp'] = event['@timestamp']"
}
}
grok {
match => [
"timestamp", "^(?<app_log_time>%{YEAR}-%{MONTHNUM}-%{MONTHDAY})"
]
}
mutate {
rename => {
"app_log_time" => "[@metadata][app_log_time]"
}
}
}
output {
if [env] == "prod" or [env] == "common" {
elasticsearch_java {
#For daily index creation used the time notation, Remove if not required.
index => "jabong-%{env}-%{app}-%{iver}-%{[@metadata][app_log_time]}"
cluster => "elasticsearch"
network_host => "172.16.84.230"
hosts => ["172.16.84.230:9300"]
protocol => "transport"
}
file {
path => "/var/log/shop/%{env}/%{app}/%{app}_%{host}_%{[@metadata][app_log_time]}.log"
}
stdout { codec => rubydebug }
}
}
的自定义模板： -
jabong-*我使用的是{
"order": 0,
"template": "jabong-*",
"settings": {
"index.refresh_interval": "5s"
},
"mappings": {
"_default_": {
"dynamic_templates": [
{
"string_fields": {
"mapping": {
"index": "analyzed",
"omit_norms": true,
"type": "string",
"fields": {
"raw": {
"ignore_above": 256,
"index": "not_analyzed",
"type": "string"
}
}
},
"match_mapping_type": "string",
"match": "*"
}
}
],
"_all": {
"omit_norms": true,
"enabled": true
},
"properties": {
"geoip": {
"dynamic": true,
"type": "object",
"properties": {
"location": {
"type": "geo_point"
}
}
},
"@version": {
"index": "not_analyzed",
"type": "string"
}
}
}
},
"aliases": {}
}
，Kafka 0.8插件位于logstash-input-kafka，而Elasticsearch的版本为2.0.8。
如果我以详细模式启动logstash，则logstash似乎停留在以下消息中： -
1.7.1有人能让我知道出了什么问题吗？