如何在Pandas中的时间序列中检测间隙和连续周期

时间:2016-10-18 21:04:14

标签: python pandas

我有一个由Date编制索引的pandas Dataframe。我希望按期间连续选择所有连续的差距,按期间连续选择所有连续的天数。我怎么能这样做?

没有列但具有日期索引的Dataframe示例:

In [29]: import pandas as pd

In [30]: dates = pd.to_datetime(['2016-09-19 10:23:03', '2016-08-03 10:53:39','2016-09-05 11:11:30', '2016-09-05 11:10:46','2016-09-05 10:53:39'])

In [31]: ts = pd.DataFrame(index=dates)

如您所见,2016-08-03和2016-09-19 差距。我如何检测这些,以便我可以创建描述性统计数据,即40个缺口,中间缺口持续时间为“x”等。另外,我可以看到 2016-09-05和2016-09-06是两个日期。我如何检测这些并打印描述性统计数据?

理想情况下,结果将在每种情况下作为另一个Dataframe返回,因为我希望使用Dataframe中的其他列来进行分组。

2 个答案:

答案 0 :(得分:5)

Pandas版本1.0.1具有内置方法DataFrame.diff(),您可以使用该方法来完成此操作。好处之一是您可以使用mean()之类的pandas系列函数来快速计算gaps系列对象的摘要统计信息

from datetime import datetime, timedelta
import pandas as pd

# Construct dummy dataframe
dates = pd.to_datetime([
    '2016-08-03',
    '2016-08-04',
    '2016-08-05',
    '2016-08-17',
    '2016-09-05',
    '2016-09-06',
    '2016-09-07',
    '2016-09-19'])
df = pd.DataFrame(dates, columns=['date'])

# Take the diff of the first column (drop 1st row since it's undefined)
deltas = df['date'].diff()[1:]

# Filter diffs (here days > 1, but could be seconds, hours, etc)
gaps = deltas[deltas > timedelta(days=1)]

# Print results
print(f'{len(gaps)} gaps with average gap duration: {gaps.mean()}')
for i, g in gaps.iteritems():
    gap_start = df['date'][i - 1]
    print(f'Start: {datetime.strftime(gap_start, "%Y-%m-%d")} | '
          f'Duration: {str(g.to_pytimedelta())}')

答案 1 :(得分:1)

这是开始的事情:

df = pd.DataFrame(np.ones(5),columns = ['ones'])
df.index = pd.DatetimeIndex(['2016-09-19 10:23:03', '2016-08-03 10:53:39', '2016-09-05 11:11:30', '2016-09-05 11:10:46', '2016-09-06 10:53:39'])
daily_rng = pd.date_range('2016-08-03 00:00:00', periods=48, freq='D')
daily_rng = daily_rng.append(df.index)
daily_rng = sorted(daily_rng)
df =  df.reindex(daily_rng).fillna(0)
df = df.astype(int)
df['ones'] = df.cumsum()

cumsum()在'ones'上创建一个分组变量,在您提供的点上对数据进行分区。如果你打印df说电子表格就有意义了:

print df.head()

                     ones
2016-08-03 00:00:00     0
2016-08-03 10:53:39     1
2016-08-04 00:00:00     1
2016-08-05 00:00:00     1
2016-08-06 00:00:00     1

print df.tail()
                     ones
2016-09-16 00:00:00     4
2016-09-17 00:00:00     4
2016-09-18 00:00:00     4
2016-09-19 00:00:00     4
2016-09-19 10:23:03     5

现在完成:

df = df.reset_index()
df = df.groupby(['ones']).aggregate({'ones':{'gaps':'count'},'index':{'first_spotted':'min'}})
df.columns = df.columns.droplevel()

给出:

              first_time  gaps
ones                          
0    2016-08-03 00:00:00     1
1    2016-08-03 10:53:39    34
2    2016-09-05 11:10:46     1
3    2016-09-05 11:11:30     2
4    2016-09-06 10:53:39    14
5    2016-09-19 10:23:03     1