Question

我尝试使用python create filter作为日志文件，如

 Thu Oct  4 23:14:40 2012 [pid 16901] CONNECT: Client "66.249.74.228"
 Thu Oct  4 23:14:40 2012 [pid 16900] [ftp] OK LOGIN: Client "66.249.74.228", anon     password "googlebot@google.com"
 Thu Oct  4 23:17:42 2012 [pid 16902] [ftp] FAIL DOWNLOAD: Client "66.249.74.228",   "/pub/10.5524/100001_101000/100039/Assembly-2011/Pa9a_assembly_config4.scafSeq.gz",  14811136 bytes, 79.99Kbyte/sec
 Fri Oct  5 00:04:13 2012 [pid 25809] CONNECT: Client "66.249.74.228"
 Fri Oct  5 00:04:14 2012 [pid 25808] [ftp] OK LOGIN: Client "66.249.74.228", anon password "googlebot@google.com"
 Fri Oct  5 00:07:16 2012 [pid 25810] [ftp] FAIL DOWNLOAD: Client "66.249.74.228", "/pub/10.5524/100001_101000/100027/Raw_data/PHOlcpDABDWABPE/090715_I80_FC427DJAAXX_L8_PHOlcpDABDWABPE_1.fq.gz", 14811136 bytes, 79.99Kbyte/sec
 Fri Oct  5 00:13:19 2012 [pid 27354] CONNECT: Client "1.202.186.53"
 Fri Oct  5 00:13:19 2012 [pid 27353] [ftp] OK LOGIN: Client "1.202.186.53", anon password "mozilla@example.com"
 Fri Oct  5 00:13:33 2012 [pid 27355] [ftp] FAIL DOWNLOAD: Client "1.202.186.53", "/pub", 0.00Kbyte/sec
 Fri Oct  5 00:26:04 2012 [pid 341] [ftp] OK DOWNLOAD: Client "210.72.156.68", "/pub/10.5524/100001_101000/100030/RNA-Seq/Mgo_2.fq.gz", 1985229528 bytes, 85.87Kbyte/sec
 Fri Oct  5 00:55:45 2012 [pid 2766] CONNECT: Client "157.82.250.217"
 Fri Oct  5 00:55:45 2012 [pid 2765] [ftp] OK LOGIN: Client "157.82.250.217", anon password "mozilla@example.com"
 Fri Oct  5 00:56:05 2012 [pid 2767] [ftp] FAIL DOWNLOAD: Client "157.82.250.217", "/pub/10.5524/100001_101000/100036/Gene_catalogue/Gene_catalogue.pep", 1638400 bytes, 81.81Kbyte/sec
 Fri Oct  5 00:57:27 2012 [pid 3056] CONNECT: Client "157.82.250.217"
 Fri Oct  5 00:57:27 2012 [pid 3055] [ftp] OK LOGIN: Client "157.82.250.217", anon password "-wget@"

日志文件有一些机器人访问记录，因此如何使用python过滤器实现真人访问记录。我已经构建了一个过滤器来获取每周记录，所以你可以帮我把它添加到里面。

import time
f= open("/opt/CLiMB/Storage1/log/vsftp.log")
def OnlyRecent(line):
    if  time.strptime(line.split("[")[0].strip(),"%a %b %d %H:%M:%S %Y")>  time.gmtime(time.time()-(60*60*24*7)): 
        return True
    return False
filename= time.strftime('%Y%m%d')+'.log'
f1= open(filename,'w')
for line in f:
    if OnlyRecent(line):
            print line
            f1.write(line)
f.close()
f1.close()

Answer 1

如果您确定使用您的系统的客户端实际上是通过查看他的密码（ googlebot@google.com 看起来像一个真正的机器人）的机器人，那么您可以只拆分一个字符串并查看第二部分是否包含机器人电子邮件：

# Add additional robot e-mails here
robot_emails = ["googlebot@google.com"]

def isRobotRecord(line):

    for email in robot_emails:
        if email in line.split("Client")[1]:
            return True
        else:
            return False

Answer 2

您可以按某个标识符对事件进行分组。我想到了pid，但似乎你的日志中的所有行都有不同的pid。您可以为每个组使用IP地址，并在找到CONNECT: Client "[IP]"时启动新组，但如果某些IP地址客户端一次有多个会话，则会失败。如果没有会话标识符，很难确定哪些行用作一个会话（组）。

当您对事件进行分组时，对于每个组，您必须检查此事件中是否由机器人留下“签名”："anon password "googlebot@google.com"

Python过滤日志文件

2 个答案: