提供以下logstash管道:
input
{
generator
{
lines => [
'{"name" : "search", "product" : { "module" : "search" , "name" : "api"}, "data" : { "query" : "toto"}}',
'{"name" : "user_interaction", "product" : { "module" : "search" , "name" : "front"}, "data" : { "query" : "toto"}}',
'{"name" : "search", "product" : { "module" : "search" , "name" : "api"}, "data" : { "query" : "toto"}}',
'{"hello": "world"}',
'{"name" :"wrong data", "data" : "I am wrong !"}',
'{"name" :"wrong data", "data" : { "hello" : "world" }}'
]
codec => json
count => 1
}
}
filter
{
mutate
{
remove_field => ["sequence", "host", "@version"]
}
}
output
{
elasticsearch
{
hosts => ["elasticsearch:9200"]
index => "events-dev6-test"
document_type => "_doc"
manage_template => false
}
stdout
{
codec => rubydebug
}
}
elasticsearch对此索引有严格的映射,因此,某些事件会产生400错误"mapping set to strict, dynamic introduction of [hello] within [data] is not allowed"
(这是正常的)。
如何在其他地方发送失败的事件(文本日志或其他弹性搜索索引)(所以我没有丢失事件)?
答案 0 :(得分:1)
Logstash 6.2引入了Dead Letter Queues,可用于执行您想要的操作。您需要在dead_letter_queue.enable: true
中启用logstash.yml
。
然后将其作为输入处理:
input {
dead_letter_queue {
path => "/path/to/data/dead_letter_queue"
commit_offsets => true
pipeline_id => "main"
}
}
output {
file {
path => ...
codec => line { format => "%{message}"}
}
}
在6.2之前,我不相信有办法做你想做的事。