在AWS

时间：2016-06-06 18:53:20

标签： ubuntu amazon-web-services fluentd

实际上我们计划在节点服务器上使用流畅的数据来获取日志并将这些日志转发到我们的日志收集器（主Fluentd服务器）然后我们可以将日志传输到s3存储桶现在我们可以将日志写入我的s3但是我在调​​试模式下重启td-agent时收到错误。

流畅的节点配置： - 

root@ip-172-31-32-115:/var/log/td-agent# cat /etc/td-agent/td-agent.conf
#Apache Access logs
<source>
  @type tail
  format apache2
  path /var/log/apache2/access.log
  pos_file /var/log/td-agent/apache2.access.pos
  tag apache.access
</source>
#Log forwardin
<match apache.access>
  @type forward
  send_timeout 60s
  recover_wait 10s
  heartbeat_interval 1s
  phi_threshold 16
  hard_timeout 60s
#primary host

  <server>
    host xx.xx.xx.xx
    port 24224
  </server>
  flush_interval 60s
</match>

日志转发器日志

root@ip-172-31-34-201:/home/ubuntu# td-agent -v 
2016-06-06 18:21:44 +0000 [info]: fluent/supervisor.rb:457:read_config: reading config file path="/etc/td-agent/td-agent.conf"
2016-06-06 18:21:44 +0000 [info]: fluent/supervisor.rb:323:supervise: starting fluentd-0.12.20
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-config-placeholders' version '0.3.1'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-mongo' version '0.7.12'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.4'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-s3' version '0.6.5'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-scribe' version '0.10.14'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td' version '0.10.28'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluent-plugin-webhdfs' version '0.4.1'
2016-06-06 18:21:44 +0000 [info]: fluent/engine.rb:114:block in configure: gem 'fluentd' version '0.12.20'
2016-06-06 18:21:44 +0000 [info]: fluent/agent.rb:125:add_match: adding match pattern="apache2.." type="forward"
2016-06-06 18:21:44 +0000 [info]: plugin/out_forward.rb:141:block in configure: adding forwarding server '172.31.33.196:24224' host="172.31.33.196" port=24224 weight=60 plugin_id="object:11e9668"
2016-06-06 18:21:44 +0000 [info]: fluent/root_agent.rb:143:add_source: adding source type="tail"
2016-06-06 18:21:44 +0000 [info]: fluent/root_agent.rb:143:add_source: adding source type="tail" plugin/out_forward.rb:232:rebuild_weight_array: rebuilding weight array lost_weight=0
2016-06-06 18:21:44 +0000 [info]: plugin/in_tail.rb:499:initialize: following tail of /var/log/apache2/access.log

log Aggregator config： - 

#Input
<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>
# S3 config
<match s3.*.*>
    @type s3

    aws_key_id AKIERPE6JxxxKXSG3GWSA
    aws_sec_key 31KnCOJdQrxxxrXEw0dYY5RqV9zegTedfrere25F1Hsvw
    s3_bucket rana.sahu
    s3_region us-west-2
    path logs/
    buffer_type           file
    buffer_path           /var/log/td-agent/s3/buffer/td
    buffer_chunk_limit    2m
    buffer_queue_limit    4098
    retry_wait            1s
    max_retry_wait        5s
    disable_retry_limit   true


    time_slice_format %Y%m%d%H
    time_slice_wait 10m
    utc

</match>

它坚持这件事

当我启动td-agent -v时，我收到此错误 2016-06-07 07:11:00 +0000 [info]：plugin / in_forward.rb：81：listen：听流畅的套接字0.0.0.0:24224 2016-06-07 07:11:00 +0000 [错误]：fluent / engine.rb：196：运行中的救援：意外错误error_class = Errno :: EADDRINUSE错误=＃   2016-06-07 07:11:00 +0000 [错误]：fluent / supervisor.rb：148：启动时阻止：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool .io-1.4.2 / lib / cool.io / server.rb:57:in initialize' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.2/lib/cool.io/server.rb:57:in new＆＃39;   2016-06-07 07:11:00 +0000 [错误]：fluent / supervisor.rb：148：启动时阻止：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool .io-1.4.2 / lib / cool.io / server.rb:57:in initialize' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_forward.rb:82:in new＆＃39;   2016-06-07 07:11:00 +0000 [错误]：fluent / supervisor.rb：148：启动时阻止：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20 / lib / fluent / plugin / in_forward.rb:82:in listen' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/plugin/in_forward.rb:51:in start＆＃39;   2016-06-07 07:11:00 +0000 [错误]：fluent / supervisor.rb：148：启动时阻止：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20 / lib / fluent / root_agent.rb:111：block in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/root_agent.rb:110:in每个＆＃39;   2016-06-07 07:11:00 +0000 [错误]：fluent / supervisor.rb：148：启动时阻止：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20 / lib / fluent / root_agent.rb:110：在start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/engine.rb:225:in开始＆＃39;   2016-06-07 07:11:00 +0000 [错误]：fluent / supervisor.rb：148：启动时阻止：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20 / lib / fluent / engine.rb：175：在run' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:597:in run_engine＆＃39;   2016-06-07 07:11:00 +0000 [错误]：fluent / supervisor.rb：148：启动时阻止：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20 / lib / fluent / supervisor.rb:148:in block in start' 2016-06-07 07:11:00 +0000 [error]: fluent/supervisor.rb:148:block in start: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/supervisor.rb:352:in来电＆＃39;   2016-06-07 07:11:00 +0000 [错误]：fluent / supervisor.rb：148：启动时阻止：/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd -0.12.20 / lib / fluent / supervisor.rb：352：在`main_process＆＃39;

如果您有任何解决方案，请告诉我。

1 个答案:

答案 0 :(得分：0)

我得到了答案，通过这些配置，我们可以将apache2日志转发到日志收集器并将它们存储到s3存储桶中。

谢谢大家。

相关问题
最新问题