Question

有没有办法删除过去30天内没有写过任何内容的所有AWS日志组？

或者反过来说，获取过去30天内没有写过任何内容的日志组列表？

Answer 1

这是我写的一些快速脚本：

#!/usr/bin/python3

# describe log groups
# describe log streams
# get log groups with the lastEventTimestamp after some time
# delete those log groups
# have a dry run option
# support profile

# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.describe_log_streams
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.describe_log_groups
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/logs.html#CloudWatchLogs.Client.delete_log_group

import boto3
import time
millis = int(round(time.time() * 1000))

delete = False
debug = False
log_group_prefix='/' # NEED TO CHANGE THESE

days = 30
# Create CloudWatchLogs client
cloudwatch_logs = boto3.client('logs')

log_groups=[]
# List log groups through the pagination interface
paginator = cloudwatch_logs.get_paginator('describe_log_groups')
for response in paginator.paginate(logGroupNamePrefix=log_group_prefix):
    for log_group in response['logGroups']:
        log_groups.append(log_group['logGroupName'])

if debug:
    print(log_groups)

old_log_groups=[]
empty_log_groups=[]
for log_group in log_groups:
    response = cloudwatch_logs.describe_log_streams(
        logGroupName=log_group, #logStreamNamePrefix='',
        orderBy='LastEventTime',
        descending=True,
        limit=1
    )
    # The time of the most recent log event in the log stream in CloudWatch Logs. This number is expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.
    if len(response['logStreams']) > 0:
        if debug:
            print("full response is:")
            print(response)
            print("Last event is:")
            print(response['logStreams'][0]['lastEventTimestamp'])
            print("current millis is:")
            print(millis)
        if response['logStreams'][0]['lastEventTimestamp'] < millis - (days * 24 * 60 * 60 * 1000):
            old_log_groups.append(log_group)
    else:
        empty_log_groups.append(log_group)

# delete log group
if delete:
    for log_group in old_log_groups:
        response = cloudwatch_logs.delete_log_group(logGroupName=log_group)
    #for log_group in empty_log_groups:
    #    response = cloudwatch_logs.delete_log_group(logGroupName=log_group)
else:
    print("old log groups are:")
    print(old_log_groups)
    print("Number of log groups:")
    print(len(old_log_groups))
    print("empty log groups are:")
    print(empty_log_groups)

Answer 2

我做了相同的设置，我想从Cloudwatch日志流中删除X天之前的日志流。

remove.py

import optparse
import sys
import os
import json
import datetime

def deletefunc(loggroupname,days):
    os.system("aws logs describe-log-streams --log-group-name {} --order-by LastEventTime > test.json".format(loggroupname))
    oldstream=[]
    milli= days * 24 * 60 * 60 * 1000
    with open('test.json') as json_file:
       data = json.load(json_file)
       for p in data['logStreams']:
         subtract=p['creationTime']+milli
         sub1=subtract/1000
         sub2=datetime.datetime.fromtimestamp(sub1).strftime('%Y-%m-%d')
         op=p['creationTime']/1000
         original=datetime.datetime.fromtimestamp(op).strftime('%Y-%m-%d')
         name=p['logStreamName']
         if original < sub2:
             oldstream.append(name)

    for i in oldstream:
       os.system("aws logs delete-log-stream --log-group-name {} --log-stream-name {}".format(loggroupname,i))


parser = optparse.OptionParser()
parser.add_option('--log-group-name', action="store", dest="loggroupname", help="LogGroupName For Eg: testing-vpc-flow-logs",type="string")
parser.add_option('-d','--days', action="store", dest="days", help="Type No.of Days in Integer to Delete Logs other than days provided",type="int")
options, args = parser.parse_args()

if options.loggroupname is None:
        print ("Provide log group name to continue.\n")
        cmd = 'python3 ' + sys.argv[0] + ' -h'
        os.system(cmd)
        sys.exit(0)

if options.days is None:
        print ("Provide date to continue.\n")
        cmd = 'python3 ' + sys.argv[0] + ' -h'
        os.system(cmd)
        sys.exit(0)


elif options.days and options.loggroupname:

    loggroupname = options.loggroupname
    days = options.days
    deletefunc(loggroupname,days)

您可以使用以下命令运行此文件：

python3 remove.py --log-group-name=testing-vpc-flow-logs --days=7

Answer 3

我不知道一种简单的方法，但你可以使用awscli（或最好是python / boto3）来描述-log-groups，然后为每个日志组调用describe-log-streams，然后为每个日志组/流对，调用get-log-events，其中--start-time为30天前。如果日志组的所有事件数组的并集为空，则表示您可以删除该日志组。

Answer 4

我一直在使用aws-cloudwatch-log-clean，可以说效果很好。

您需要安装boto3，然后：

./sweep_log_streams.py [log_group_name]

它有一个--dry-run选项，可让您检查一下您的期望。

请注意，如果您在ECS中有一个长时间运行的进程，而该进程在日志中处于静默状态，并且由于日志保留期而导致CW中的日志已被截断为空。删除其空日志流可能会中断并挂起服务，因为它无处可发布日志至...

删除在X天内未被触及的AWS日志组

4 个答案: