我有一个CSV文件,其中包含多天的每一分钟的行数。它由数据采集系统生成,有时会错过几行。
数据看起来像这样 - 日期时间字段后跟一些整数
"2017-01-07 03:00:02","7","3","2","13","0"
"2017-01-07 03:01:02","7","3","2","13","0"
"2017-01-07 03:02:02","7","3","2","12","0"
"2017-01-07 03:07:02","7","3","2","12","0"
"2017-01-07 03:08:02","6","3","2","12","1"
"2017-01-07 03:09:02","7","3","2","12","0"
"2017-01-07 03:10:02","6","3","2","11","1"
上面(实际数据)示例中缺少行。由于样本之间的数据变化不大,我只想将最后的有效数据复制到缺失的行中。我遇到的问题是检测哪些行丢失。
我正在使用我拼凑在一起的python程序处理CSV(我对python很新)。 这可以处理我拥有的数据。
import csv
import datetime
with open("minutedata.csv", 'rb') as f:
reader = csv.reader(f, delimiter=',')
for row in reader:
date = datetime.datetime.strptime (row [0],"%Y-%m-%d %H:%M:%S")
v1 = int(row[1])
v2 = int(row[2])
v3 = int(row[3])
v4 = int(row[4])
v5 = int(row[5])
...(process values)...
...(save data)...
我不确定如何检查当前行是按顺序排列,还是在丢失一些行之后。
编辑添加:
我正在尝试使用Pandas,感谢jeremycg指向它。
我在CSV中添加了标题行,现在它看起来像:
time,v1,v2,v3,v4,v5
"2017-01-07 03:00:02","7","3","2","13","0"
"2017-01-07 03:01:02","7","3","2","13","0"
"2017-01-07 03:02:02","7","3","2","12","0"
"2017-01-07 03:07:02","7","3","2","12","0"
"2017-01-07 03:08:02","6","3","2","12","1"
"2017-01-07 03:09:02","7","3","2","12","0"
"2017-01-07 03:10:02","6","3","2","11","1"
处理代码现在是:
import pandas as pd
import io
z = pd.read_csv('minutedata.csv')
z['time'] = pd.to_datetime(z['time'])
z.set_index('time').reindex(pd.date_range(min(z['time']), max(z['time']),freq="1min")).ffill()
for row in z:
date = datetime.datetime.strptime (row [0],"%Y-%m-%d %H:%M:%S")
v1 = int(row[1])
v2 = int(row[2])
v3 = int(row[3])
v4 = int(row[4])
v5 = int(row[5])
...(process values)...
...(save data)...
但是出错了:
Traceback (most recent call last):
File "process_day.py", line 14, in <module>
z.set_index('time').reindex(pd.date_range(min(z['time']), max(z['time']), freq="1min")).ffill()
File "/usr/local/lib/python2.7/site-packages/pandas/core/frame.py", line 2821, in reindex
**kwargs)
File "/usr/local/lib/python2.7/site-packages/pandas/core/generic.py", line 2259, in reindex fill_value, copy).__finalize__(self)
File "/usr/local/lib/python2.7/site-packages/pandas/core/frame.py", line 2767, in _reindex_axes
fill_value, limit, tolerance)
File "/usr/local/lib/python2.7/site-packages/pandas/core/frame.py", line 2778, in _reindex_index allow_dups=False)
File "/usr/local/lib/python2.7/site-packages/pandas/core/generic.py", line 2371, in _reindex_with_indexers copy=copy)
File "/usr/local/lib/python2.7/site-packages/pandas/core/internals.py", line 3839, in reindex_indexer self.axes[axis]._can_reindex(indexer)
File "/usr/local/lib/python2.7/site-packages/pandas/indexes/base.py", line 2494, in _can_reindex raise ValueError("cannot reindex from a duplicate axis")
ValueError: cannot reindex from a duplicate axis
我迷失了现在声称被打破的东西。
请参阅下面的评论,了解此修复程序。
现在的工作代码是:
import pandas as pd
import datetime
z = pd.read_csv('minutedata1.csv')
z = z[~z.time.duplicated()]
z['time'] = pd.to_datetime(z['time'])
z.set_index('time').reindex(pd.date_range(min(z['time']), max(z['time']),freq="1min")).ffill()
for index,row in z.iterrows():
date = datetime.datetime.strptime (row [0],"%Y-%m-%d %H:%M:%S")
v1 = int(row[1])
v2 = int(row[2])
v3 = int(row[3])
v4 = int(row[4])
v5 = int(row[5])
...(process values)...
...(save data)...
衷心感谢所有帮助过的人。 - 大卫
答案 0 :(得分:2)
你可能应该使用pandas,因为它是为这种东西制作的。
首先阅读csv:
import pandas as pd
import io
x = '''
time,a,b,c,d,e
"2017-01-07 03:00:02","7","3","2","13","0"
"2017-01-07 03:01:02","7","3","2","13","0"
"2017-01-07 03:02:02","7","3","2","12","0"
"2017-01-07 03:07:02","7","3","2","12","0"
"2017-01-07 03:08:02","6","3","2","12","1"
"2017-01-07 03:09:02","7","3","2","12","0"
"2017-01-07 03:10:02","6","3","2","11","1"''' #your data, with added headers
z = pd.read_csv(io.StringIO(x)) #you can use your file name here
现在z是一个pandas数据帧:
z.head()
time a b c d e
0 2017-01-07 03:00:02 7 3 2 13 0
1 2017-01-07 03:01:02 7 3 2 13 0
2 2017-01-07 03:02:02 7 3 2 12 0
3 2017-01-07 03:07:02 7 3 2 12 0
4 2017-01-07 03:08:02 6 3 2 12 1
我们希望: 将'time'列转换为pd.datetime:
z['time'] = pd.to_datetime(z['time'])
将数据框的'index'设置为时间,然后重新索引我们的范围:
z = z.set_index('time').reindex(pd.date_range(min(z['time']), max(z['time']), freq="1min"))
z
a b c d e
2017-01-07 03:00:02 7.0 3.0 2.0 13.0 0.0
2017-01-07 03:01:02 7.0 3.0 2.0 13.0 0.0
2017-01-07 03:02:02 7.0 3.0 2.0 12.0 0.0
2017-01-07 03:03:02 NaN NaN NaN NaN NaN
2017-01-07 03:04:02 NaN NaN NaN NaN NaN
2017-01-07 03:05:02 NaN NaN NaN NaN NaN
2017-01-07 03:06:02 NaN NaN NaN NaN NaN
2017-01-07 03:07:02 7.0 3.0 2.0 12.0 0.0
2017-01-07 03:08:02 6.0 3.0 2.0 12.0 1.0
2017-01-07 03:09:02 7.0 3.0 2.0 12.0 0.0
2017-01-07 03:10:02 6.0 3.0 2.0 11.0 1.0
然后使用.ffill()从前一个值填写:
z.ffill()
a b c d e
2017-01-07 03:00:02 7.0 3.0 2.0 13.0 0.0
2017-01-07 03:01:02 7.0 3.0 2.0 13.0 0.0
2017-01-07 03:02:02 7.0 3.0 2.0 12.0 0.0
2017-01-07 03:03:02 7.0 3.0 2.0 12.0 0.0
2017-01-07 03:04:02 7.0 3.0 2.0 12.0 0.0
2017-01-07 03:05:02 7.0 3.0 2.0 12.0 0.0
2017-01-07 03:06:02 7.0 3.0 2.0 12.0 0.0
2017-01-07 03:07:02 7.0 3.0 2.0 12.0 0.0
2017-01-07 03:08:02 6.0 3.0 2.0 12.0 1.0
2017-01-07 03:09:02 7.0 3.0 2.0 12.0 0.0
2017-01-07 03:10:02 6.0 3.0 2.0 11.0 1.0
或者,所有在一起:
z = pd.read_csv(io.StringIO(x))
z['time'] = pd.to_datetime(z['time'])
z.set_index('time').reindex(pd.date_range(min(z['time']), max(z['time']), freq="1min")).ffill()
答案 1 :(得分:2)
建议使用jeremycg建议使用的pandas。虽然如果你正在寻找没有大熊猫的解决方案,那么它就是:
import csv
import datetime
data = []
with open("minutedata.csv", newline='') as f:
reader = csv.reader(f, delimiter=',')
prev_date = None
for row in reader:
date = datetime.datetime.strptime(row[0], "%Y-%m-%d %H:%M:%S")
if prev_date:
diff = date - prev_date
if diff > datetime.timedelta(minutes=1):
for i in range((int(diff.total_seconds() / 60) - 1)):
new_date = prev_date + datetime.timedelta(minutes=i + 1)
new_row = [str(new_date)] + row[1:]
data.append(",".join(new_row))
prev_date = date
data.append(",".join(row))
print(data)
说明: 我们遍历每一行并检查当前行的日期与前一行的日期
diff = date - prev_date
如果我们发现差异超过1分钟,我们会输入一个循环,该循环针对缺失数据的范围运行
if diff > datetime.timedelta(minutes=1):
for i in range((int(diff.total_seconds() / 60) - 1)):
...
我们通过在上一个日期添加分钟来添加计算缺失值
new_date = prev_date + datetime.timedelta(minutes=i + 1)
new_row = [str(new_date)] + row[1:]
你完成了!