我正在尝试使用logstash（v5.4.3）和logstash-output-s3（5.0.7）将kubernetes日志发送到s3。 logstash正在创建logstash程序化访问测试对象，而不是实际的日志文件。以下是我的输入和输出配置。
我正在尝试使用logstash（v5.4.3）和logstash-output-s3（5.0.7）将kubernetes日志发送到s3。 logstash正在创建logstash程序化访问测试对象，而不是实际的日志文件。以下是我的输入和输出配置。
input {
file {
# kubelet created symlinks to docker container logs
path => "/var/log/containers/*.log"
tags => ["kubernetes", "docker"]
sincedb_path => "/var/log/containers/sincedb-kubernetes"
}
}
output {
if "kubernetes" and "docker" in [tags] {
s3{
access_key_id => "XXXXXXXXXX"
secret_access_key => "XXXXXXXXXXXXXXXXXXXXXXXX"
region => "us-east-1"
bucket => "nonapp-logs"
size_file => 2048
time_file => 5
prefix => "non-app-logs-%{+YYYY.MM.dd}"
canned_acl => "private"
server_side_encryption => true
}
}
}
下面是我启动服务时的日志
[2019-06-19T11:49:52,648][DEBUG][logstash.agent ] starting agent
[2019-06-19T11:49:52,649][DEBUG][logstash.agent ] starting pipeline {:id=>"main"}
[2019-06-19T11:49:53,355][DEBUG][logstash.outputs.s3 ] Start periodic rotation check
[2019-06-19T11:49:53,364][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2019-06-19T11:49:53,775][INFO ][logstash.pipeline ] Pipeline main started
[2019-06-19T11:49:53,806][DEBUG][logstash.agent ] Starting puma
[2019-06-19T11:49:53,807][DEBUG][logstash.agent ] Trying to start WebServer {:port=>9600}
[2019-06-19T11:49:53,811][DEBUG][logstash.api.service ] [api-service] start
[2019-06-19T11:49:53,818][DEBUG][logstash.inputs.file ] _globbed_files: /var/log/containers/*.log: glob is: ["/var/log/containers/file1-42861859-dx660_default_file1-6faea70b0e43f4805378b02fcef43008c8a815b560bb6517b9e888a3622c24f1.log", "/var/log/containers/file2-n6dkl_monitoring_file2-26964aa335e8f60d7ed2e3fe1673346e4a33e322e1d5d42162b22a5b77f4f766.log", "/var/log/containers/aqua-agent-zck5m_file3-5d523fef15c1d4fc7979b11a33b39cb4e290b6a33b87c6f7789ed6be4849bd6a.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-98810bf8d063e535cd51f10be8b50a61a07922b5dd3ed8d20298f9af68dc56a2.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-48cd2f69df073460bbd0799ce9d918bfa4ac9587517e25b6cea9664939db4d59.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-5e7ff1e609f7cb858aeb17267d43c7f0602c6f79a9aef34ec7e0f0762f7bf4b1.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-98b868ffbc6d463092f634632040203ea3a8c349f667ff66d26562f52e4503b7.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-beeecf6ad217620a86f0ce5fb77820300df5a9a5019d0d5c1042987f37e6981d.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-021fef919f6892a5b343752851b96161b6f975a8457754f32a1ab9bf2d8d07da.log", "/var/log/containers/file2-n6dkl_monitoring_file2-1708ecd0708f31eaac2c0906a1cf0044152de4c750e2ba10fd56d32c7026779f.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-55209391560c6e544a4eacb54cc7d19588f7b263663a182da5092c5edb3911cf.log", "/var/log/containers/logstash-9k41j_kube-system_logstash-0577c19fe06f0de92a88b6f74a42313169945ca730f9f6c0c4aeb8de736bfccc.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-48a5402aa102d30d4b61c93130cb7501deb8e67620b17c68b65e713d9b6a4003.log", "/var/log/containers/file5-532757934-9vh43_default_file5-14d5f783f8f9836703006dff011ec68e8e84187a1027555c524c761087594a6f.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-demo-api-e3e65ab389eec9302aeb76ce34c0b08dc68088df7a3e19be7daee8e43fc32a93.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-ng-demo-2b678c4ef7bab59eb686fa6284d60477f5e26aa877c53e1efe72e2c297df87e9.log"]
[2019-06-19T11:49:53,901][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2019-06-19T11:49:58,808][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2019-06-19T11:50:03,811][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2019-06-19T11:50:07,823][DEBUG][logstash.inputs.file ] _globbed_files: /var/log/containers/*.log: glob is: ["/var/log/containers/file1-42861859-dx660_default_file1-6faea70b0e43f4805378b02fcef43008c8a815b560bb6517b9e888a3622c24f1.log", "/var/log/containers/file2-n6dkl_monitoring_file2-26964aa335e8f60d7ed2e3fe1673346e4a33e322e1d5d42162b22a5b77f4f766.log", "/var/log/containers/aqua-agent-zck5m_file3-5d523fef15c1d4fc7979b11a33b39cb4e290b6a33b87c6f7789ed6be4849bd6a.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-98810bf8d063e535cd51f10be8b50a61a07922b5dd3ed8d20298f9af68dc56a2.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-48cd2f69df073460bbd0799ce9d918bfa4ac9587517e25b6cea9664939db4d59.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-5e7ff1e609f7cb858aeb17267d43c7f0602c6f79a9aef34ec7e0f0762f7bf4b1.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-98b868ffbc6d463092f634632040203ea3a8c349f667ff66d26562f52e4503b7.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-beeecf6ad217620a86f0ce5fb77820300df5a9a5019d0d5c1042987f37e6981d.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-021fef919f6892a5b343752851b96161b6f975a8457754f32a1ab9bf2d8d07da.log", "/var/log/containers/file2-n6dkl_monitoring_file2-1708ecd0708f31eaac2c0906a1cf0044152de4c750e2ba10fd56d32c7026779f.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-55209391560c6e544a4eacb54cc7d19588f7b263663a182da5092c5edb3911cf.log", "/var/log/containers/logstash-9k41j_kube-system_logstash-0577c19fe06f0de92a88b6f74a42313169945ca730f9f6c0c4aeb8de736bfccc.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-48a5402aa102d30d4b61c93130cb7501deb8e67620b17c68b65e713d9b6a4003.log", "/var/log/containers/file5-532757934-9vh43_default_file5-14d5f783f8f9836703006dff011ec68e8e84187a1027555c524c761087594a6f.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-demo-api-e3e65ab389eec9302aeb76ce34c0b08dc68088df7a3e19be7daee8e43fc32a93.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-ng-demo-2b678c4ef7bab59eb686fa6284d60477f5e26aa877c53e1efe72e2c297df87e9.log"]
[2019-06-19T11:50:08,362][DEBUG][logstash.outputs.s3 ] Periodic check for stale files
[2019-06-19T11:50:08,813][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2019-06-19T11:50:13,813][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2019-06-19T11:50:18,814][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2019-06-19T11:50:22,831][DEBUG][logstash.inputs.file ] _globbed_files: /var/log/containers/*.log: glob is: ["/var/log/containers/file1-42861859-dx660_default_file1-6faea70b0e43f4805378b02fcef43008c8a815b560bb6517b9e888a3622c24f1.log", "/var/log/containers/file2-n6dkl_monitoring_file2-26964aa335e8f60d7ed2e3fe1673346e4a33e322e1d5d42162b22a5b77f4f766.log", "/var/log/containers/aqua-agent-zck5m_file3file3-5d523fef15c1d4fc7979b11a33b39cb4e290b6a33b87c6f7789ed6be4849bd6a.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-98810bf8d063e535cd51f10be8b50a61a07922b5dd3ed8d20298f9af68dc56a2.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-48cd2f69df073460bbd0799ce9d918bfa4ac9587517e25b6cea9664939db4d59.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-5e7ff1e609f7cb858aeb17267d43c7f0602c6f79a9aef34ec7e0f0762f7bf4b1.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-98b868ffbc6d463092f634632040203ea3a8c349f667ff66d26562f52e4503b7.log", "/var/log/containers/kube-proxy-ip-172-31-61-232.ec2.internal_kube-system_kube-proxy-beeecf6ad217620a86f0ce5fb77820300df5a9a5019d0d5c1042987f37e6981d.log", "/var/log/containers/file4-1989789293-s8t08_aquasec_file4-021fef919f6892a5b343752851b96161b6f975a8457754f32a1ab9bf2d8d07da.log", "/var/log/containers/file2-n6dkl_monitoring_file2-1708ecd0708f31eaac2c0906a1cf0044152de4c750e2ba10fd56d32c7026779f.log", "/var/log/containers/example-node-2999102209-cd279_default_example-node-55209391560c6e544a4eacb54cc7d19588f7b263663a182da5092c5edb3911cf.log", "/var/log/containers/logstash-9k41j_kube-system_logstash-0577c19fe06f0de92a88b6f74a42313169945ca730f9f6c0c4aeb8de736bfccc.log", "/var/log/containers/calico-node-jlb5x_kube-system_calico-node-48a5402aa102d30d4b61c93130cb7501deb8e67620b17c68b65e713d9b6a4003.log", "/var/log/containers/file5-532757934-9vh43_default_file5-14d5f783f8f9836703006dff011ec68e8e84187a1027555c524c761087594a6f.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-demo-api-e3e65ab389eec9302aeb76ce34c0b08dc68088df7a3e19be7daee8e43fc32a93.log", "/var/log/containers/file6-ng-demo-1167756042-19pzx_default_file6-ng-demo-2b678c4ef7bab59eb686fa6284d60477f5e26aa877c53e1efe72e2c297df87e9.log"]
[2019-06-19T11:50:23,366][DEBUG][logstash.outputs.s3 ] Periodic check for stale files
[2019-06-19T11:50:23,814][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
^C[2019-06-19T11:50:23,887][WARN ][logstash.runner ] SIGINT received. Shutting down the agent.
[2019-06-19T11:50:23,889][DEBUG][logstash.instrument.periodicpoller.os] PeriodicPoller: Stopping
[2019-06-19T11:50:23,889][DEBUG][logstash.instrument.periodicpoller.jvm] PeriodicPoller: Stopping
[2019-06-19T11:50:23,889][DEBUG][logstash.instrument.periodicpoller.persistentqueue] PeriodicPoller: Stopping
[2019-06-19T11:50:23,892][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2019-06-19T11:50:23,894][DEBUG][logstash.pipeline ] Closing inputs
[2019-06-19T11:50:23,894][DEBUG][logstash.inputs.file ] stopping {:plugin=>"LogStash::Inputs::File"}
[2019-06-19T11:50:23,894][DEBUG][logstash.pipeline ] Closed inputs
[2019-06-19T11:50:24,850][DEBUG][logstash.inputs.file ] closing {:plugin=>"LogStash::Inputs::File"}
[2019-06-19T11:50:24,851][DEBUG][logstash.pipeline ] Input plugins stopped! Will shutdown filter/output workers.
[2019-06-19T11:50:24,918][DEBUG][logstash.pipeline ] Pushing flush onto pipeline
[2019-06-19T11:50:24,919][DEBUG][logstash.pipeline ] Pushing shutdown {:thread=>"#<Thread:0x624b35a1 run>"}
[2019-06-19T11:50:24,919][DEBUG][logstash.pipeline ] Pushing shutdown {:thread=>"#<Thread:0x1e3416c8 sleep>"}
[2019-06-19T11:50:24,921][DEBUG][logstash.pipeline ] Shutdown waiting for worker thread #<Thread:0x624b35a1>
[2019-06-19T11:50:24,955][DEBUG][logstash.pipeline ] Shutdown waiting for worker thread #<Thread:0x1e3416c8>
[2019-06-19T11:50:24,956][DEBUG][logstash.outputs.s3 ] closing {:plugin=>"LogStash::Outputs::S3"}
[2019-06-19T11:50:24,956][DEBUG][logstash.outputs.s3 ] Uploading current workspace
[2019-06-19T11:50:24,961][DEBUG][logstash.pipeline ] Pipeline main has been shutdown
enter code here