Cloudtrail日志记录到AWS Elasticsearch

时间:2018-10-16 22:29:35

标签: amazon-web-services elasticsearch logstash kibana-6 amazon-cloudtrail

试图将多个AWS帐户的cloudtrail日志从s3转换为elasticsearch,直到现在一切都停止了,事情似乎一直在进行中。错误显示如下所示

[2018-10-16T21:33:42,096][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>8}
[2018-10-16T21:33:44,406][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443/, :path=>"/"}
[2018-10-16T21:33:44,430][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443/"}
[2018-10-16T21:33:51,426][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>413, :url=>"https://vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443/_bulk"}

这也是我的logstash配置,就像使用logstash进行提取一样

```

    input {
       s3 {
         bucket => "dummy-s3"
         region => "eu-west-1"
         type => "cloudtrail"
         sincedb_path => "/tmp/logstash/cloudtrail"
         exclude_pattern => "/CloudTrail-Digest/"
         interval => 120
         codec => "json"
       }
    }

    filter {
        if [type] == "cloudtrail" {
            json {
                source => "message"
            }
            split {
                    field => "Records"
                    add_tag => "splitted"

            }
            if ("splitted" in [tags]) {
              date {
                    match => ["eventTime", "ISO8601"]
                    remove_tag => ["splitted"]
                    remove_field => ["timestamp"]
                   }
            }

            geoip {
                source => "[Records][sourceIPAddress]"
                target => "geoip"
                add_tag => ["cloudtrail-geoip"]
            }

            mutate {
                 gsub => [

                  "eventSource", "\.amazonaws\.com$", "",
                  "apiVersion", "_", "-"
            ]

            }
        }
    }

    output {
        elasticsearch {
          hosts => ["vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443"]
          ssl => true
          index => "cloudtrail-%{+YYYY.MM.dd}"
          doc_as_upsert => true
          template_overwrite => true
          }
        stdout {
         codec => rubydebug
        }
     }
}

当日志存储开始或从ubuntu ec2日志重新启动并摄取几分钟后,便会停止

任何帮助将不胜感激。

0 个答案:

没有答案
相关问题