按时间间隔生成输出,其中源中没有值

时间:2017-10-26 08:46:02

标签: logparser

我有一个消息日志文件,每个消息发送一行,时间戳舍入为分钟。我可以使用Logparser生成每分钟消息频率的报告,并输出显示相同的图表。这是我用来生成报告的查询:

SELECT  
    QUANTIZE (Date, 60) AS M, 
    COUNT(*) AS Total 
FROM '[LOGFILEPATH]' 
GROUP BY M 
ORDER BY M

这非常适合在生成流量时显示那些分钟的消息流量速率。

但是,我想输出一个图表,该图表还显示消息计数 0 的分钟间隔 - 在条形图中,该分钟栏将为0。这将显示没有消息流量的时间间隔。

当然,这不在源数据中,因为它没有记录“没有消息”。任何想法如何做到这一点?我实际上并不关心它是否是Logparser - Powershell是我很乐意使用的东西。甚至是Excel,虽然我非常喜欢可以在没有人工干预的情况下编写脚本的解决方案。

我们是Windows商店,对开发工具的访问权限有限,所以请不要使用Python,bash等解决方案。

这里有一些示例(真实的,混淆的)数据,这些数据接近半小时的间隔,没有记录流量。

Date,MID,Host,Sender,Recipient,Subject,Last State
2017-10-26 01:56,1078941,smtp3 (10.0.0.156),bounces@example.com,2.O@d.com.au,Notification has been created successfully,Message 1078941 to 2.O@d.com.au received remote SMTP response 'ok'
2017-10-26 01:56,1078938,smtp3 (10.0.0.156),bounces@example.com,a@y.com.au,Notification has been created successfully,Message 1078940 to bounces@example.com received remote SMTP response '2.0.0'
2017-10-26 01:55,4571113,smtp1 (10.0.1.59),bounces@example.com,K@c.com,Notification has been created successfully,Message 4571113 to K@c.com received remote SMTP response '2.0.0'
2017-10-26 01:55,1078936,smtp3 (10.0.0.156),bounces@example.com,G.F@y.com.au,Notification has been created successfully,Message 1078936 to G.F@y.com.au received remote SMTP response 'ok'
2017-10-26 01:54,4571083,smtp1 (10.0.1.59),bounces@example.com,B.H@b.com,Notification has been created successfully,Message 4571083 to B.H@b.com received remote SMTP response 'Mail accepted'.
2017-10-26 01:53,1078927,smtp3 (10.0.0.156),bounces@example.com,S.R@g.com,Notification has been created successfully,Message 1078927 to S.R@g.com received remote SMTP response '2.0.0'
2017-10-26 01:52,4571051,smtp1 (10.0.1.59),bounces@example.com,A.U@r.com.au,Notification has been created successfully,Message 4571051 to A.U@r.com.au received remote SMTP response 'ok'
2017-10-26 01:23,4570598,smtp1 (10.0.1.59),bounces@example.com,T@h.com,Notification has been created successfully,Message 4570598 to T@h.com received remote SMTP response '2.0.0'
2017-10-26 01:23,4570594,smtp1 (10.0.1.59),bounces@example.com,I@k.com.au,Notification has been created successfully,Message 4570594 to I@k.com.au received remote SMTP response 'ok'
2017-10-26 01:22,4570579,smtp1 (10.0.1.59),bounces@example.com,I.C@s.com.au,Notification has been created successfully,Message 4570579 to I.C@s.com.au received remote SMTP response '2.0.0'
2017-10-26 01:22,4570577,smtp1 (10.0.1.59),bounces@example.com,P.M@h.net.au,Notification has been created successfully,Message 4570577 to P.M@h.net.au received remote SMTP response '2.0.0'
2017-10-26 01:22,4570575,smtp1 (10.0.1.59),bounces@example.com,O@h.com.au,Notification has been created successfully,Message 4570575 to O@h.com.au received remote SMTP response '2.0.0'
2017-10-26 01:21,4570571,smtp1 (10.0.1.59),bounces@example.com,O@f.com,Notification has been created successfully,Message 4570571 to O@f.com received remote SMTP response 'ok'
2017-10-26 01:21,4570557,smtp1 (10.0.1.59),bounces@example.com,O.M@m.v.edu.au,Notification has been created successfully,Message 4570557 to O.M@m.v.edu.au received remote SMTP response 'OK'
2017-10-26 01:21,4570549,smtp1 (10.0.1.59),bounces@example.com,A@a.com,Notification has been created successfully,Message 4570550 to bounces@example.com received remote SMTP response '2.0.0'
2017-10-26 01:20,1078803,smtp3 (10.0.0.156),bounces@example.com,M.1@l.com.au,Notification has been created successfully,Message 1078803 to M.1@l.com.au received remote SMTP response '2.0.0'
2017-10-26 01:20,1078802,smtp3 (10.0.0.156),bounces@example.com,B@g.com,New Account,Message 1078802 to B@g.com received remote SMTP response '2.0.0'
2017-10-26 01:20,4570539,smtp1 (10.0.1.59),bounces@example.com,I@v.com.au,Notification has been created successfully,Message 4570539 to I@v.com.au received remote SMTP response 'OK'
2017-10-26 01:19,1078794,smtp3 (10.0.0.156),bounces@example.com,A.T@b.com,Notification has been created successfully,Message 1078794 to A.T@b.com received remote SMTP response 'ok'

1 个答案:

答案 0 :(得分:1)

正如评论中所提到的,这将是一个多步骤的过程,从获取日志文件中的日期开始:

SELECT MIN(Date) AS MinDate, MAX(Date) as MaxDate 
INTO theDates.csv 
FROM yourlog.log

theDates.csv 中包含的日期中的读数使用Powershell程序来生成与您的日志格式匹配且每分钟仅包含1条记录的日志文件。

然后您可以稍微调整一下运行原始查询:

SELECT 
   QUANTIZE(Date, 60) as M,
   SUB(COUNT(*), 1) as total
FROM yourlog.log, yourfakelogfile.log
ORDER BY M
GROUP BY M

轻微的调整是计数减1,这样你就可以在没有活动的时间段内得到零。

将所有这三个步骤放在一个PowerShell脚本中,您就拥有了一个自动化,可重复的过程。