如何配置Fluentd配置文件以将日志发送到Kafka?

时间:2019-05-23 09:26:57

标签: apache-kafka rubygems centos7 fluentd cassandra-2.1

我一直在努力创建一个syslog应用程序,在该应用程序中,我必须收集使用Fluentd生成的各种日志,并将它们发送给Cassandra,以便稍后进行分析。我必须使用Kafka作为一种缓冲区或流处理器。要在Fluentd上使用Kafka,我必须配置Fluentd。

我已经安装了Fluentd,Cassandra,Kafka和Zookeeper,它们都独立运行,没有任何错误。但是,当我尝试使用fluent.conf文件将Kafka连接至流利时,我遇到了多个错误。

我的fluent.conf文件是:

<source>
  @type syslog
  bind 0.0.0.0
  port 5140
  tag default
  protocol_type udp
  source_hostname_key source_host
  <parse>
   message_format auto
  </parse>
</source>

<match **>
  @type stdout
</match>

<match app.**>
  @type kafka_buffered
  zookeeper             localhost:2181
  zookeeper_path        root/attackfence/kafka_2.11-2.2.0/zookeeeper
  partition_key         partition
  partition_key_key     partition_key
  message_key_key       message_key
  default_topic         nil
  default_partition_key nil
  default_message_key   nil
  output_data_type      json
  output_include_tag    false
  output_include_time   false
  exclude_topic_key     false
  exclude_partition_key false
  get_kafka_client_log  true
</match>

我得到的错误是:

2019-05-23 18:22:44 +0530 [error]: #0 unexpected error error_class=NoMethodError error="undefined method `each' for nil:NilClass"
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluent-plugin-kafka-0.9.4/lib/fluent/plugin/out_kafka_buffered.rb:117:in `refresh_client'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluent-plugin-kafka-0.9.4/lib/fluent/plugin/out_kafka_buffered.rb:208:in `start'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/compat/call_super_mixin.rb:42:in `start'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:165:in `block in start'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:153:in `each'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:153:in `block in lifecycle'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:140:in `each'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:140:in `lifecycle'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/root_agent.rb:164:in `start'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/engine.rb:274:in `start'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/engine.rb:219:in `run'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:799:in `run_engine'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:549:in `block in run_worker'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:724:in `main_process'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/supervisor.rb:544:in `run_worker'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/rubies/ruby-2.6.0/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/gems/fluentd-1.3.3/bin/fluentd:8:in `<top (required)>'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/bin/fluentd:23:in `load'
  2019-05-23 18:22:44 +0530 [error]: #0 /usr/local/rvm/gems/ruby-2.6.0/bin/fluentd:23:in `<main>'

0 个答案:

没有答案