Pandas,检查datetimeindex的重采样30分钟时间段中是否存在时间戳值

时间:2015-11-05 18:15:10

标签: python pandas

我在pandas中创建了一个带有datetimeindex的重采样数据帧(DF1)。我有一个单独的数据框(DF2),其中包含datetimeindextime列。如果来自DF2的time实例落在DF1中datetimeindex的30分钟区间内。我想用DF1中30分钟的bin中的适当time标记DF2中speed的每个实例。

DF1

                   boat_id      speed
time                                      
2015-01-13 09:00:00   28.000000   0.000000
2015-01-13 09:30:00   28.000000   0.723503
2015-01-13 10:00:00   28.000000   2.239399

DF2

                      id  boat_id                 time  state     
time                                                                          
2015-01-18 16:09:03   319437       28  2015-01-18 16:09:03      2    
2015-01-18 16:18:43   319451       28  2015-01-18 16:18:43      0    
2015-03-01 09:39:51   507108       31  2015-03-01 09:39:51      1    
2015-03-01 09:40:58   507109       31  2015-03-01 09:40:58      0 

期望的结果

                      id  boat_id                 time      state   speed
time                                                                          
2015-01-18 16:09:03   319437       28  2015-01-18 16:09:03      2 nan   
2015-01-18 16:18:43   319451       28  2015-01-18 16:18:43      0 nan   
2015-03-01 09:39:51   507108       31  2015-03-01 09:39:51      1 2.239399   
2015-03-01 09:40:58   507109       31  2015-03-01 09:40:58      0 2.239399

我创建了这个脚本来尝试这样做,但我认为它失败了,因为DF1的datetimeindex是不可变的,因此我的timedelta请求不会为块创建一个起始点。我有一个想法是,是否有可能将DF1的datetimeindex复制到一个新的列中,其中对象是可变的,但我还没有管理它,所以我不能100%确定逻辑。我很高兴修补,但此刻我已经停滞了一段时间,所以希望别人可能会有一些想法。提前谢谢。

for row in DF1.iterrows():
    for dfrow in DF2.iterrows():
        if dfrow[0] > row[0] - dt.timedelta(minutes=30) and dfrow[0] < row[0]:
            df['test'] =  row[1]

2 个答案:

答案 0 :(得分:1)

迭代的性能非常低。更好的是使用矢量化解决方案。我使用了两次函数mergeDocs

输入:

print df1
                     boat_id     speed
time                                  
2015-03-01 09:00:00       28  0.000000
2015-03-01 09:30:00       28  0.723503
2015-03-01 10:00:00       28  2.239399

print df2
                         id  boat_id                time  state
time                                                           
2015-01-18 16:09:03  319437       28 2015-01-18 16:09:03      2
2015-01-18 16:18:43  319451       28 2015-01-18 16:18:43      0
2015-03-01 09:39:51  507108       31 2015-03-01 09:39:51      1
2015-03-01 09:40:58  507109       31 2015-03-01 09:40:58      0

我重置两个数据框的索引并创建由i填充的辅助列1

df1 = df1.reset_index()
df2 = df2.reset_index(drop=True)
df1['i'] =  df2['i'] = 1
print df1
                 time  boat_id     speed  i
0 2015-03-01 09:00:00       28  0.000000  1
1 2015-03-01 09:30:00       28  0.723503  1
2 2015-03-01 10:00:00       28  2.239399  1
print df2
       id  boat_id                time  state  i
0  319437       28 2015-01-18 16:09:03      2  1
1  319451       28 2015-01-18 16:18:43      0  1
2  507108       31 2015-03-01 09:39:51      1  1
3  507109       31 2015-03-01 09:40:58      0  1

然后我通过辅助列i合并了两个数据帧。

df = df2.merge(df1, on='i', how='left')
df = df.rename(columns={'time_y':'Bin_time', 'time_x':'time'})
print df
        id  boat_id_x                time  state  i            Bin_time  \
0   319437         28 2015-01-18 16:09:03      2  1 2015-03-01 09:00:00   
1   319437         28 2015-01-18 16:09:03      2  1 2015-03-01 09:30:00   
2   319437         28 2015-01-18 16:09:03      2  1 2015-03-01 10:00:00   
3   319451         28 2015-01-18 16:18:43      0  1 2015-03-01 09:00:00   
4   319451         28 2015-01-18 16:18:43      0  1 2015-03-01 09:30:00   
5   319451         28 2015-01-18 16:18:43      0  1 2015-03-01 10:00:00   
6   507108         31 2015-03-01 09:39:51      1  1 2015-03-01 09:00:00   
7   507108         31 2015-03-01 09:39:51      1  1 2015-03-01 09:30:00   
8   507108         31 2015-03-01 09:39:51      1  1 2015-03-01 10:00:00   
9   507109         31 2015-03-01 09:40:58      0  1 2015-03-01 09:00:00   
10  507109         31 2015-03-01 09:40:58      0  1 2015-03-01 09:30:00   
11  507109         31 2015-03-01 09:40:58      0  1 2015-03-01 10:00:00   

    boat_id_y     speed  
0          28  0.000000  
1          28  0.723503  
2          28  2.239399  
3          28  0.000000  
4          28  0.723503  
5          28  2.239399  
6          28  0.000000  
7          28  0.723503  
8          28  2.239399  
9          28  0.000000  
10         28  0.723503  
11         28  2.239399  

输出按bin时间过滤:

df = df[((df.time >= (df.Bin_time - dt.timedelta(minutes=30))) & (df.time <= df.Bin_time ))]
df = df.drop(['Bin_time', 'id', 'boat_id_x', 'boat_id_y','state', 'i' ], axis=1 )
print df
                  time     speed
8  2015-03-01 09:39:51  2.239399
11 2015-03-01 09:40:58  2.239399

dftime列与数据框df2合并。

df = df2.merge(df, on='time', how='left').reset_index(drop=True)
df = df.drop([ 'i' ], axis=1 )
print df
       id  boat_id                time  state     speed
0  319437       28 2015-01-18 16:09:03      2       NaN
1  319451       28 2015-01-18 16:18:43      0       NaN
2  507108       31 2015-03-01 09:39:51      1  2.239399
3  507109       31 2015-03-01 09:40:58      0  2.239399

比较矢量化和索引方法,您可以在类似的答案here中找到。

答案 1 :(得分:0)

改编:

for i in range(1, len(DF1.index)):
    DF2.between_time(DF1.index[i-1], DF1.index[i], include_start=True,
                     include_end=True).loc[:,'speed'] = DF1.loc[DF1.index[i],'speed']

how to use dataframe between_time() function

的帮助下