Question

在AWS中构建了在RHEL 7.2上运行的ELK服务器。计划是让它从S3桶中摄取cloudtrail日志，然后在kibana前端做魔术，但是它没有工作，我已经失去了生命中的日子，因此为什么我＆＃ 39;我伸手去寻求帮助。我可以从我的logstash.log中看到它正在读取S3存储桶中的文件，但这是关于它的。似乎没有其他事情发生。

我的设置：

RHEL 7.2
kibana-4.5.0-1.x86_64
logstash-2.3.2-1.noarch
elasticsearch-2.3.3-1.noarch
nginx-1.6.3-8.el7.x86_64 (reverse proxy kibana to port 80)


This is what my /etc/logstash/conf.d looks like :
-rw-r--r-- 1 root root 574 May 31 14:55 02-cloudtrail-input.conf
-rw-r--r-- 1 root root 432 May 31 15:04 30-elasticsearch-output.conf

root @ elk conf.d] #cat *

input {   
    s3 
    bucket => "xyz..cloudtrail"
        access_key_id => 'XYZ'
        secret_access_key => 'ABC'
        delete => false
        codec => "cloudtrail"
        prefix => "cloudtrail/AWSLogs/xxxxx/CloudTrail/ap-southeast-2/2016/"
        type => "cloudtrail"
        interval => 10 # seconds
        region => "ap-southeast-2"
        sincedb_path => "/data/logstash/cloudtrail/db/sincedb"
    }
}
output {
        #stdout {}
        stdout { codec => rubydebug }
    elasticsearch {
        hosts => "localhost:9200"
        sniffing => true
        #codec => "cloudtrail"
        #index => "cloudtrail"
        index => "logstash-%{+YYYY.MM.dd}"
        #index => "%{[@metadata][cloudtrail]}-%{+YYYY.MM.dd}"
        #index => "cloudtrail-%{+YYYY.MM.dd}"
        action => create
        manage_template => false
    workers => 4
    }
}

installed plugins :
logstash-codec-cloudtrail
logstash-input-s3
logstash-output-s3

我不知道如何在logstash输出中定义索引，但我可以在kibana前端搜索它并选择3个时间帧选项中的一个但不知道这意味着什么。我应该嗅探=真吗？我应该定义行动=＆gt;创建？我是否应该在logstash输入和输出中定义cloudtrail编解码器？

任何人都可以告诉我它的含义是什么＆＃34;插件没有在命名空间中定义，检查插件文件＆＃34; ？虽然安装了所有必需的插件（我认为），但听起来我觉得它无法找到插件

以下是我手动启动logstash时的输出...但它对我来说意义不大..

Reading config file {:config_file=>"/etc/logstash/conf.d/02-cloudtrail-input.conf", :level=>:debug, :file=>"logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
Reading config file {:config_file=>"/etc/logstash/conf.d/30-elasticsearch-output.conf", :level=>:debug, :file=>"logstash/config/loader.rb", :line=>"69", :method=>"local_config"}
Plugin not defined in namespace, checking for plugin file {:type=>"input", :name=>"s3", :path=>"logstash/inputs/s3", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"codec", :name=>"cloudtrail", :path=>"logstash/codecs/cloudtrail", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
config LogStash::Codecs::CloudTrail/@spool_size = 50 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@bucket = "abcdbase-cloudtrail" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@access_key_id = "XYZ" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@secret_access_key = "ABC" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@delete = false {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@codec = <LogStash::Codecs::CloudTrail spool_size=>50> {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@prefix = "abcdbase-trail/AWSLogs/554658506446/CloudTrail/ap-southeast-2/2016/" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@type = "cloudtrail" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@interval = 10 {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@region = "ap-southeast-2" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@sincedb_path = "/data/logstash/cloudtrail/db/sincedb" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@add_field = {} {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@use_ssl = true {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@credentials = [] {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@backup_to_bucket = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@backup_add_prefix = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@backup_to_dir = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@exclude_pattern = nil {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
config LogStash::Inputs::S3/@temporary_directory = "/tmp/logstash" {:level=>:debug, :file=>"logstash/config/mixin.rb", :line=>"153", :method=>"config_init"}
Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"stdout", :path=>"logstash/outputs/stdout", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}
Plugin not defined in namespace, checking for plugin file {:type=>"output", :name=>"elasticsearch", :path=>"logstash/outputs/elasticsearch", :level=>:debug, :file=>"logstash/plugin.rb", :line=>"76", :method=>"lookup"}

当logstash.log继续说method =＆gt;＆＃34; list_new_files＆＃34;时，这是什么意思，这是否意味着它忙于重新读取S3存储桶（每次logstash停止/启动时））？

我知道很多问题，但我想在将此设置放入垃圾箱之前我会尝试一些帮助。

感谢

Answer 1

下面的配置对我有用。

input {
s3 {
  access_key_id => "xxxx"
  bucket => "my-bucket"
  region => "us-east-1"
  secret_access_key => "xxx"
  prefix => "your prefix"
  type => "s3"
  add_field => { source => gzfiles }
  codec => cloudtrail {}
}
}

output {
  elasticsearch {
        hosts => localhost
    }
}

我安装了下面提到的插件 -

logstash-codec-cloudtrail logstash-input-s3

logstash - 输出 - elasticsearc

有关更多插件详细信息，您可以触发插件列表命令 -

/ opt / logstash / bin / plugin list

Linux ELK服务器读取cloudtrail日志 - 故障排除问题

我的设置：

1 个答案: