我使用logstash来消耗kafka数据并将日志保存到hdfs,并且我希望使用不同的kafka主题拆分hdfs输出,我的logstash配置如下:
input {
kafka {
bootstrap_servers => "192.168.1.14:9092"
group_id => "logstash_act"
topics => ["act", "log"]
decorate_events => true
consumer_threads => 5
}
}
output {
if [kafka][topic] == "act" {
webhdfs {
host => "192.168.1.8"
port => 50070
user => "root"
path => "/ubuntu/input/logstash-%{+YYYY}-%{+MM}-%{+dd}.act"
codec => plain { format => "%{message}" }
idle_flush_time => 10
retry_interval => 0.5 flush_size => 500
}
}
else if [kafka][topic] == "log" {
webhdfs {
host => "192.168.1.8"
port => 50070
user => "root"
path => "/ubuntu/input/logstash-%{+YYYY}-%{+MM}-%{+dd}.log"
codec => plain { format => "%{message}" }
idle_flush_time => 10
retry_interval => 0.5 flush_size => 500
}
}
}
没有错误,但是根本没有输出。