Question

我有一个精通+弹性搜索+ kibana设置。 Fluentd配置为使用此插件输出到弹性搜索： https://github.com/uken/fluent-plugin-elasticsearch。问题是，当它正在运行时，fluentd永远不会将其缓冲区刷新到弹性搜索，它只是将数据存储在内存缓冲区中。发出关闭后，刷新缓冲区并更新弹性搜索。

尽管设置了flush_interval 60s和buffer_chunk_limit 25k，但仍会出现此问题。默认值也会产生同样的问题。

流利的收集器日志文件输出和cfg：

    2014-07-11 11:45:44 +0100 [info]: fluent/supervisor.rb:202:supervise: starting fluentd-0.10.48
2014-07-11 11:45:44 +0100 [info]: fluent/supervisor.rb:304:read_config: reading config file path="/etc/fluent/fluent.conf"
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered buffer plugin 'file'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered buffer plugin 'memory'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'debug_agent'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'exec'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'forward'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'gc_stat'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'http'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'monitor_agent'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'object_space'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'status'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'tcp'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'unix'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'syslog'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'tail'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'copy'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'exec'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'exec_filter'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'file'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'forward'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'null'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'roundrobin'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'stdout'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'tcp'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'unix'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'test'
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-elasticsearch' version '0.3.0'
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:87:block in configure: gem 'fluentd' version '0.10.48'
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:91:configure: using configuration file: <ROOT>
  <source>
    type forward
    port 24224
    bind 0.0.0.0
  </source>
  <source>
    type syslog
    port 5544
    protocol_type tcp
    bind 0.0.0.0
    tag index.syslog
  </source>
  <match index.**>
    type elasticsearch
    logstash_format true
    host localhost
    port 9200
    include_tag_key true
    tag_key tag
  </match>
</ROOT>
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:101:block in configure: adding source type="forward"
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:101:block in configure: adding source type="syslog"
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:117:block in configure: adding match pattern="index.**" type="elasticsearch"
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'elasticsearch'
2014-07-11 11:45:44 +0100 [info]: plugin/in_forward.rb:75:listen: listening fluent socket on 0.0.0.0:24224
2014-07-11 11:45:45 +0100 [debug]: plugin/in_syslog.rb:183:listen: listening syslog socket on 0.0.0.0:5544 with tcp
2014-07-11 11:45:45 +0100 [trace]: plugin/in_forward.rb:189:initialize: accepted fluent socket from '10.18.80.20:57119': object_id=70254255494700
2014-07-11 11:45:45 +0100 [trace]: plugin/in_forward.rb:245:on_close: closed fluent socket object_id=70254255494700
2014-07-11 11:45:46 +0100 [trace]: plugin/in_forward.rb:189:initialize: accepted fluent socket from '10.18.80.20:57120': object_id=70254256323480
2014-07-11 11:45:46 +0100 [trace]: plugin/in_forward.rb:245:on_close: closed fluent socket object_id=70254256323480
2014-07-11 11:45:47 +0100 [trace]: plugin/in_forward.rb:189:initialize: accepted fluent socket from '10.18.80.20:57121': object_id=70254256319280
2014-07-11 11:45:47 +0100 [trace]: plugin/in_forward.rb:245:on_close: closed fluent socket object_id=70254256319280
.......

任何人都可以帮助解释为什么流利的人不会急于进行弹性搜索？感谢。

Answer 1

经过多次故障排除后，我发现安装的特定版本的ruby中存在一个错误，导致线程在启动时出现死锁。关闭时没有出现问题，因为它被称为不同的。这是一个特定的bug red ruby 2.0.0-p353;下一个版本ruby 2.0.0-p451解决了这个问题。

https://groups.google.com/forum/#!msg/fluentd/t2uKfttiYCw/P8zy5kZpEIIJ

Answer 2

/ var / spool / fluentd下有流畅的缓冲区吗？他们成长了吗？

您可以查看ElasticSearch日志吗？

grep --color FrameException /var/log/elasticsearch/elasticsearch.log
org.elasticsearch.common.netty.handler.codec.frame.TooLongFrameException: HTTP content length exceeded 655360000 bytes.

Answer 3

查看粘贴的标准输出，看起来不像为out_elasticsearch插件设置了“flush_interval”：

<match index.**>
  type elasticsearch
  logstash_format true
  host localhost
  port 9200
  include_tag_key true
  tag_key tag
</match>

您可以尝试在上面的配置中插入一行“flush_interval 60s”，等待一分钟左右，看看数据是否出现在Elasticsearch中？

Fluentd不会将任何数据刷新到弹性搜索 - 但会在关闭时刷新

3 个答案: