我流利的gem是将日志从日志文件发送到ES和S3存储桶。
我正在使用hypok_role_credentials选项提供对S3存储桶的访问。日志已成功发送到ES,但出现访问被拒绝错误(见下文)
这是在无法访问互联网的AWS EC2实例上实现的。安全已拒绝使用aws_key_id和aws_sec_key的选项。
IAM角色已经附加到EC2实例,并且可以使用aws cli命令的--put-object选项从该实例创建对象。
我的fluent.conf是:
<source>
@type tail
path /opt/apigee/var/log/messages/*MsgG.log, /opt/apigee/var/log/messages/*MsgS.log
pos_file /var/log/fluentd/application.log.pos
tag application
rotate_wait 7
read_from_head true
emit_unmatched_lines true
<parse>
@type json
</parse>
</source>
<source>
@type tail
path /opt/apigee/var/log/messages/*Audit.log
pos_file /var/log/fluentd/audit.log.pos
tag audit
rotate_wait 7
read_from_head true
emit_unmatched_lines true
<parse>
@type json
</parse>
</source>
# Store Data in Elasticsearch and S3
<match application>
@type copy
<store>
@type elasticsearch
hosts host1:port1,host2:port2
logstash_format true
logstash_prefix prefix
logstash_prefix_separator _
logstash_dateformat %Y-%m-%d
reconnect_on_error true
reload_on_failure true
flush_interval 1s
</store>
<store>
@type s3
<assume_role_credentials>
role_arn arn:aws:iam::xxxxx:role/role
role_session_name session-name
</assume_role_credentials>
s3_bucket bucket-name
s3_region eu-west-2
s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
path path-path
time_slice_format %Y-%m-%d
<buffer>
@type file
path /var/lib/fluentd/application-s3-buffer
flush_interval 10s
retry_wait 1
retry_type periodic
retry_timeout 72h
</buffer>
</store>
</match>
<match audit>
@type copy
<store>
@type elasticsearch
hosts host1:port1,host2:port2
logstash_format true
logstash_prefix prefix
logstash_prefix_separator _
logstash_dateformat %Y-%m-%d
reconnect_on_error true
reload_on_failure true
flush_interval 1s
</store>
<store>
@type s3
<assume_role_credentials>
role_arn arn:aws:iam::xxxxx:role/role
role_session_name session-name
</assume_role_credentials>
s3_bucket bucket-name
s3_region eu-west-2
s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
path cop_audit_
time_slice_format %Y-%m-%d
<buffer>
@type file
path /var/lib/fluentd/audit-s3-buffer
flush_interval 10s
retry_wait 1
retry_type periodic
retry_timeout 72h
</buffer>
</store>
</match>
我希望配置能够正常运行,因为我相信已经说明了所有必需的参数。但是,出现以下错误:
2019-10-02 11:52:25 +0100 [error]: #0 fluent/log.rb:362:error: unexpected error error_class=Aws::STS::Errors::AccessDenied error="Access denied"
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/seahorse/client/request.rb:70:in `send_request'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/aws-sdk-sts/client.rb:596:in `assume_role'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/assume_role_credentials.rb:49:in `refresh'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/refreshing_credentials.rb:20:in `initialize'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/aws-sdk-core-3.68.0/lib/aws-sdk-core/assume_role_credentials.rb:40:in `initialize'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluent-plugin-s3-1.1.11/lib/fluent/plugin/out_s3.rb:429:in `new'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluent-plugin-s3-1.1.11/lib/fluent/plugin/out_s3.rb:429:in `setup_credentials'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluent-plugin-s3-1.1.11/lib/fluent/plugin/out_s3.rb:200:in `start'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/root_agent.rb:203:in `block in start'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/root_agent.rb:191:in `each'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/root_agent.rb:191:in `block in lifecycle'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/root_agent.rb:178:in `each'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/root_agent.rb:178:in `lifecycle'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/root_agent.rb:202:in `start'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/engine.rb:274:in `start'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/engine.rb:219:in `run'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/supervisor.rb:808:in `run_engine'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/supervisor.rb:551:in `block in run_worker'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/supervisor.rb:733:in `main_process'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/supervisor.rb:546:in `run_worker'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/lib/fluent/command/fluentd.rb:320:in `<top (required)>'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /usr/lib/ruby/gems/2.6.0/gems/fluentd-1.7.2/bin/fluentd:8:in `<top (required)>'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /bin/fluentd:23:in `load'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/supervisor.rb:551:block in run_worker: /bin/fluentd:23:in `<main>'
2019-10-02 11:52:25 +0100 [error]: #0 fluent/log.rb:362:error: unexpected error error_class=Aws::STS::Errors::AccessDenied error="Access denied"
答案 0 :(得分:0)
我不需要担任角色,因为IAM角色已经附加到我的实例中。