Question

我试图通过log stash解析一些带有固定宽度字段的示例文件，然后将解析后的数据放入弹性搜索中。我不希望将重复项放入弹性搜索中并实现此目的，我正在配置操作是更新并专门指定文档ID的id字段。但是，当我启动我的日志存储时，它失败并显示错误，如下所示：

Failed action.  {:status=>404, :action=>["update", {:_id=>"RECORD_CODE", :_index=>"transactions", :_type=>"lot13", :_routing=>nil}, #<LogStash::Event:0x389f17de @metadata_accessors=#<LogStash::Util::Accessors:0x5dd5606b @store={"path"=>"/Users/priya/sample.log", "retry_count"=>0}, @lut={"[path]"=>[{"path"=>"/Users/priya/sample.log", "retry_count"=>0}, "path"]}>, @cancelled=false, @data={"message"=>"878979797978779779797978", "@version"=>"1", "@timestamp"=>"2015-11-26T12:43:06.677Z", "host"=>"priyas-MacBook-Pro.local", "path"=>"/Users/priya/sample.log", "RECORD_CODE"=>"8", "SEQUENCE_NUMBER"=>"789797", "REG_NUMBER"=>"979787797797", "DATA_TYPE"=>"979"}, @metadata={"path"=>"/Users/priya/sample.log", "retry_count"=>0}, @accessors=#<LogStash::Util::Accessors:0x78c1600a @store={"message"=>"878979797978779779797978", "@version"=>"1", "@timestamp"=>"2015-11-26T12:43:06.677Z", "host"=>"priyas-MacBook-Pro.local", "path"=>"/Users/priya/sample.log", "RECORD_CODE"=>"8", "SEQUENCE_NUMBER"=>"789797", "REG_NUMBER"=>"979787797797", "DATA_TYPE"=>"979"}, @lut={"host"=>[{"message"=>"878979797978779779797978", "@version"=>"1", "@timestamp"=>"2015-11-26T12:43:06.677Z", "host"=>"priyas-MacBook-Pro.local", "path"=>"/Users/priya/sample.log", "RECORD_CODE"=>"8", "SEQUENCE_NUMBER"=>"789797", "REG_NUMBER"=>"979787797797", "DATA_TYPE"=>"979"}, "host"], "path"=>[{"message"=>"878979797978779779797978", "@version"=>"1", "@timestamp"=>"2015-11-26T12:43:06.677Z", "host"=>"priyas-MacBook-Pro.local", "path"=>"/Users/priya/sample.log", "RECORD_CODE"=>"8", "SEQUENCE_NUMBER"=>"789797", "REG_NUMBER"=>"979787797797", "DATA_TYPE"=>"979"}, "path"], "message"=>[{"message"=>"878979797978779779797978", "@version"=>"1", "@timestamp"=>"2015-11-26T12:43:06.677Z", "host"=>"priyas-MacBook-Pro.local", "path"=>"/Users/priya/sample.log", "RECORD_CODE"=>"8", "SEQUENCE_NUMBER"=>"789797", "REG_NUMBER"=>"979787797797", "DATA_TYPE"=>"979"}, "message"], "RECORD_CODE"=>[{"message"=>"878979797978779779797978", "@version"=>"1", "@timestamp"=>"2015-11-26T12:43:06.677Z", "host"=>"priyas-MacBook-Pro.local", "path"=>"/Users/priya/sample.log", "RECORD_CODE"=>"8", "SEQUENCE_NUMBER"=>"789797", "REG_NUMBER"=>"979787797797", "DATA_TYPE"=>"979"}, "RECORD_CODE"], "SEQUENCE_NUMBER"=>[{"message"=>"878979797978779779797978", "@version"=>"1", "@timestamp"=>"2015-11-26T12:43:06.677Z", "host"=>"priyas-MacBook-Pro.local", "path"=>"/Users/priya/sample.log", "RECORD_CODE"=>"8", "SEQUENCE_NUMBER"=>"789797", "REG_NUMBER"=>"979787797797", "DATA_TYPE"=>"979"}, "SEQUENCE_NUMBER"], "REG_NUMBER"=>[{"message"=>"878979797978779779797978", "@version"=>"1", "@timestamp"=>"2015-11-26T12:43:06.677Z", "host"=>"priyas-MacBook-Pro.local", "path"=>"/Users/priya/sample.log", "RECORD_CODE"=>"8", "SEQUENCE_NUMBER"=>"789797", "REG_NUMBER"=>"979787797797", "DATA_TYPE"=>"979"}, "REG_NUMBER"], "DATA_TYPE"=>[{"message"=>"878979797978779779797978", "@version"=>"1", "@timestamp"=>"2015-11-26T12:43:06.677Z", "host"=>"priyas-MacBook-Pro.local", "path"=>"/Users/priya/sample.log", "RECORD_CODE"=>"8", "SEQUENCE_NUMBER"=>"789797", "REG_NUMBER"=>"979787797797", "DATA_TYPE"=>"979"}, "DATA_TYPE"]}>>], :response=>{"update"=>{"_index"=>"transactions", "_type"=>"lot13", "_id"=>"RECORD_CODE", "status"=>404, "error"=>{"type"=>"document_missing_exception", "reason"=>"[lot13][RECORD_CODE]: document missing", "shard"=>"-1", "index"=>"transactions"}}}, :level=>:warn}

我的配置文件的内容是：

# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
input {
    file {
    path => "/Users/priya/sample.log"
    start_position => beginning 
    sincedb_path => "/Users/priya/sample-sincedb-file"
  }
}

filter {
    grok {
        match => {"message" => "(?<RECORD_CODE>.{1})(?<SEQUENCE_NUMBER>.{6})(?<REG_NUMBER>.{12})(?<DATA_TYPE>.{3})"}
    }


}

output {
    elasticsearch {
       hosts => localhost
       index => transactions
       document_type => lot13
       document_id => RECORD_CODE
 #      template => "/Users/priya/template.json"
 #      template_name => "sample_template"
       action => update 

    }
    stdout {
    codec => rubydebug
    }
}

当我在弹性搜索中搜索时，没有检索到任何东西 - 所以很明显数据不会进入弹性搜索。有人可以帮帮我吗？

谢谢和问候，普里亚

Answer 1

这里的错误是： "error"=>{"type"=>"document_missing_exception", "reason"=>"[lot13][RECORD_CODE]: document missing", "shard"=>"-1", "index"=>"transactions"

因此，当文档丢失时更新失败。

如果Elasticsearch中不存在document_id，则需要使用doc_as_upsert = true和action => update创建包含来源的新文档。

日志存储更新操作失败，状态为404

1 个答案: