使用python计算CSV文件数据的持续时间和平均值

时间:2013-08-12 17:02:59

标签: python csv python-2.7

我有一个csv文件,我想要做的是创建一个脚本,用户输入源ip和目标IP。一旦匹配在csv文件中。它将占用用户输入的所有源和目标IP,并计算源和目标IP的用户输入的多个匹配的会话之间的时间差。最后,脚本也将执行持续时间的平均值。下面是我的csv列A数据的示例,但是csv有几个列,如Time,Source Ip和Destination IP。我们可以使用三个不同的列,而不是使用三个不同的列,我们可以使用我们需要的三个信息。

_raw

2013-07-18 04:54:15.871 UDP 172.12.332.11:20547 172.12.332.11:20547 - > 172.56.213.80:53创建忽略0

2013-07-18 04:54:15.841 UDP 192.33.230.81:37192 192.81.130.82:37192 - > 172.81.123.70:53创建忽略0

2013-07-18 04:54:15.831 TCP 172.12.332.11:42547 172.12.332.11:42547
- > 172.56.213.80:53创建忽略0

下面是我在python中的代码,它不再起作用了。现在发生的一切都是它跳过ip并且什么都不做。请帮我解决,因为我迷失了为什么它不起作用。

我在python中的代码:

import sys
from sys import argv
from datetime import datetime, timedelta

script, source, destination, filename = argv #assign the script arguments to variables
line_num = 0 #for keeping track of the current line number
count = 0 #for counting occurrences of source/destination IPs
occurrences = [] 
#array to store all of the matching occurrences of source/destination IPs

line_array = [] #array to store line numbers
avg = 0 #average
total = 0 #sum of microseconds

#function for converting timedelta to microseconds
def timedelta_to_microtime(td):
return td.microseconds + (td.seconds + td.days * 86400) * 1000000
#use 'try' to catch IOexception
try:
for line in open(filename):
        #if the first character is a number, read line

            if line[0].isdigit():
           if source and destination in line:
            #increment counter for each occurrence of matching  IP combination
            count+=1
            #get the first 23 characters from the line (the date/time)
             #and convert it to a datetime object using the "%Y-%m-%d %H:%M:%S.%f"
             #format, then add it to the array named "occurrences."
        occurrences.append(datetime.strptime(line[:23], '%Y-%m-%d %H:%M:%S.%f'))
            line_array.append(line_num)
        #if the first character is not a number, it's the headers, skip them
        else:
            line_num += 2
            continue #go to next line
        line_num += 1 #counter to keep track of line (solely for testing purposes)
#if the script can't find the data file, notify user and terminate

except IOError:
    print "\n[ERROR]: Cannot read data file, check file name and try again."
    sys.exit()

print "\nFound %s matches for [source: %s] and [destination: %s]:\n" % (len(occurrences), source, destination)

if len(occurrences) != 0: 
#if there are no occurrences, there aren't any times to show! so don't print this line
    print "Time between adjacent connections:\n"

for i in range(len(occurrences)):
if i == 0:
        continue #if it is the first slot in the array, continue to next              slot (can't  subtract from array[0-1] slot)
else:
    #find difference in timedate objects (returns difference in timedelta object)
       difference = (occurrences[i-1]-occurrences[i])
       #for displaying line numbers
       time1 = line_array[i-1]
       time2 = line_array[i]
       #convert timedelta object to microseconds for computing average
       time_m = timedelta_to_microtime(difference)
       #add current microseconds to existing microseconds
       total += time_m
       print "Line %s and Line %s: %s" % (time1, time2, difference)

#check to make sure there are things to take the average of
if len(occurrences) != 0:
    #compute average
    #line read as: total divided by the length of the occurrences array as a float
    #minus 1, divided by 1,000,000 (to convert microseconds back into seconds)
    avg = (total / float((len(occurrences)-1)))/1000000
    print "\nAverage: %s seconds" % (avg)

1 个答案:

答案 0 :(得分:1)

如果您使用像pandas这样的高级库,则可以更轻松地解决此问题。让我演示一下:

假设您在file.csv中保存了下一个数据文件:

2013-07-18 04:54:15.871 UDP 172.12.332.11:20547 172.12.332.11:20547 -> 172.56.213.80:53 CREATE Ignore 0
2013-07-18 04:54:15.841 UDP 192.33.230.81:37192 192.81.130.82:37192 -> 172.81.123.70:53 CREATE Ignore 0
2013-07-18 04:54:15.831 TCP 172.12.332.11:42547 172.12.332.11:42547 -> 172.56.213.80:53 CREATE Ignore 0
2013-07-18 04:54:15.821 UDP 192.33.230.81:37192 192.81.130.82:37192 -> 172.81.123.70:53 CREATE Ignore 0
2013-07-18 04:54:15.811 TCP 172.12.332.11:42547 172.12.332.11:42547 -> 172.56.213.80:53 CREATE Ignore 0

首先我们将其读入数据框:

>>> df = pd.read_table('file.csv', sep=' ', header=None, parse_dates=[[0,1]])
>>> print df.to_string()
                         0_1    2                    3                    4   5                 6       7       8  9
0 2013-07-18 04:54:15.871000  UDP  172.12.332.11:20547  172.12.332.11:20547  ->  172.56.213.80:53  CREATE  Ignore  0
1 2013-07-18 04:54:15.841000  UDP  192.33.230.81:37192  192.81.130.82:37192  ->  172.81.123.70:53  CREATE  Ignore  0
2 2013-07-18 04:54:15.831000  TCP  172.12.332.11:42547  172.12.332.11:42547  ->  172.56.213.80:53  CREATE  Ignore  0
3 2013-07-18 04:54:15.821000  UDP  192.33.230.81:37192  192.81.130.82:37192  ->  172.81.123.70:53  CREATE  Ignore  0
4 2013-07-18 04:54:15.811000  TCP  172.12.332.11:42547  172.12.332.11:42547  ->  172.56.213.80:53  CREATE  Ignore  0

我们只需要0_1,第4和第6列

>> df = df[['0_1', 4, 6]]
>> print df.to_string()
                         0_1                    4                 6
0 2013-07-18 04:54:15.871000  172.12.332.11:20547  172.56.213.80:53
1 2013-07-18 04:54:15.841000  192.81.130.82:37192  172.81.123.70:53
2 2013-07-18 04:54:15.831000  172.12.332.11:42547  172.56.213.80:53
3 2013-07-18 04:54:15.821000  192.81.130.82:37192  172.81.123.70:53
4 2013-07-18 04:54:15.811000  172.12.332.11:42547  172.56.213.80:53

然后我们应该修复IP地址并删除端口:

>>> df[4] = df[4].str.split(':').str.get(0)
>>> df[6] = df[6].str.split(':').str.get(0)
>>> print df.to_string()
                         0_1              4              6
0 2013-07-18 04:54:15.871000  172.12.332.11  172.56.213.80
1 2013-07-18 04:54:15.841000  192.81.130.82  172.81.123.70
2 2013-07-18 04:54:15.831000  172.12.332.11  172.56.213.80
3 2013-07-18 04:54:15.821000  192.81.130.82  172.81.123.70
4 2013-07-18 04:54:15.811000  172.12.332.11  172.56.213.80

假设您对源地址172.12.332.11和目标172.56.213.80感兴趣。我们将过滤掉那些:

>>> filtered = df[(df[4] == '172.12.332.11') & (df[6] == '172.56.213.80')]
>>> print filtered.to_string()
                         0_1              4              6
0 2013-07-18 04:54:15.871000  172.12.332.11  172.56.213.80
2 2013-07-18 04:54:15.831000  172.12.332.11  172.56.213.80
4 2013-07-18 04:54:15.811000  172.12.332.11  172.56.213.80

现在我们需要计算时间戳之间的差异:

>>> timestamps = filtered['0_1']
>>> diffs = (timestamps.shift() - timestamps).dropna()
>>> print diffs.to_string()
2   00:00:00.040000
4   00:00:00.020000

我们现在可以计算出我们想要的任何统计数据:

>>> diffs.mean() # this is in nanoseconds
30000000.0
>>> diffs.std()
14142135.62373095

修改:对于您发送给我的数据

import io
import pandas as pd

def load_dataframe(filename):
    # First you read the data as a regular csv file and extract the _raw column values
    values = pd.read_csv(filename)['_raw'].values
    # Cleanup the values: remove newline character
    values = map(lambda x: x.replace('\n', ' '), values)
    # Add them to a stream
    s = io.StringIO(u'\n'.join(values))
    # And now everithing is the same just read it from the stream
    df = pd.read_table(s, sep='\s+', header=None, parse_dates=[[0,1]])[['0_1',4, 6]]
    df[4] = df[4].str.split(':').str.get(0)
    df[6] = df[6].str.split(':').str.get(0)
    return df

def get_diffs(df, source, destination):
    timestamps = df[(df[4] == source) & (df[6] == destination)]['0_1']
    return (timestamps.shift() - timestamps).dropna()


def main():
    filename = raw_input('Enter filename: ')
    df = load_dataframe(filename)
    while True:
       source = raw_input('Enter source IP: ').strip()
       destination = raw_input('Enter destination IP: ').strip()
       diffs = get_diffs(df, source, destination)
       for i, row in enumerate(diffs):
           print('row %d - row %d = %s' % (i+2, i+1, row.astype('timedelta64[ms]')))
       print('Mean: %s' % diffs.mean())
       yn = raw_input('Again? [y/n]: ').lower().strip()
       if yn != 'y':
            return

if __name__ == '__main__':
    main()

使用示例:

$ python test.py
Enter filename: Data.csv
Enter source IP: 172.16.122.21
Enter destination IP: 172.55.102.107
Mean: 3333333.33333
Std: 5773502.6919
Again? [y/n]: n