Question

我有一些日志数据，它们代表一个项目（id）和一个动作开始的时间戳，我想确定每个项目在两次动作之间的时间。

例如，我有一些看起来像这样的数据：

data = [{"timestamp":"2019-05-21T14:17:29.265Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-21T14:21:49.722Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-21T15:16:25.695Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-21T15:16:25.696Z","id":"ff9dad92-e7c1-47a5-93a7-6e49533a6e25"},{"timestamp":"2019-05-22T07:51:17.49Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T08:11:13.948Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T11:52:59.897Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T11:53:03.406Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-22T11:53:03.481Z","id":"ff12891e-5786-438b-891c-abd4244723b4"},{"timestamp":"2019-05-21T14:23:08.147Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T14:29:18.228Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T15:17:09.831Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T15:17:09.834Z","id":"fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa"},{"timestamp":"2019-05-21T14:02:19.072Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T14:02:34.867Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T14:12:28.877Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T15:19:19.567Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T15:19:19.582Z","id":"fd3554cd-b83d-49af-a8e6-7bf41c741cd0"},{"timestamp":"2019-05-21T09:58:02.185Z","id":"f89c2e3e-06dc-467b-813b-dc92f2692f63"},{"timestamp":"2019-05-21T10:07:24.044Z","id":"f89c2e3e-06dc-467b-813b-dc92f2692f63"}]
stack = pd.DataFrame(data)
stack.head()

我尝试获取所有唯一ID来拆分数据帧，然后使用索引花费时间来与原始集合重新组合，例如，但是对于大型数据集，该功能非常慢，并且将两个索引弄乱了和时间戳顺序导致结果未命中。

import ciso8601 as time
records = []
for i in list(stack.id.unique()):
    dff = stack[stack.id == i]
    time_taken = []
    times = []
    i = 0
    for _, row in dff.iterrows():
        if bool(times):
            print(_)
            current_time = time.parse_datetime(row.timestamp)
            prev_time = times[i]
            time_taken = current_time - prev_time
            times.append(current_time)
            i+=1
            records.append(dict(index = _, time_taken = time_taken.seconds))
        else:
            records.append(dict(index = _, time_taken = 0))
            times.append(time.parse_datetime(row.timestamp))

x = pd.DataFrame(records).set_index('index')
stack.merge(x, left_index=True, right_index=True, how='inner')

是否有一个整洁的大熊猫groupby并采用了这种方式，这样我就不必拆分框架并将其存储在内存中，从而可以引用子集中的前一行？

谢谢

Answer 1

您可以使用GroupBy.diff：

stack['timestamp'] = pd.to_datetime(stack['timestamp'])
stack['timestamp']= (stack.sort_values(['id','timestamp'])
                            .groupby('id')
                            .diff()['timestamp']
                            .dt.total_seconds()
                            .round().fillna(0))

print(stack['time_taken'])
0         0.0
1       260.0
2      3276.0
3         0.0
4         0.0
5      1196.0
6     13306.0
7         4.0
8         0.0
9         0.0
10      370.0
11     2872.0
...

如果您希望按日期对结果数据框进行排序，请执行以下操作：

stack['timestamp'] = pd.to_datetime(stack['timestamp']) 
stack = stack.sort_values(['id','timestamp'])
stack['time_taken'] = (stack.groupby('id')
                            .diff()['timestamp'] 
                            .dt.total_seconds() 
                            .round()
                            .fillna(0))

Answer 2

如果不需要将时间戳替换为日期时间，请用to_datetime创建由日期时间填充的系列，然后传递给DataFrameGroupBy.diff，然后用Series.dt.total_seconds转换为秒，必要时将Series.round舍入。并将缺失的值替换为0：

t = pd.to_datetime(stack['timestamp'])
stack['time_taken'] = t.groupby(stack['id']).diff().dt.total_seconds().round().fillna(0)

print (stack)
                                      id                 timestamp  time_taken
0   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T14:17:29.265Z         0.0
1   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T14:21:49.722Z       260.0
2   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T15:16:25.695Z      3276.0
3   ff9dad92-e7c1-47a5-93a7-6e49533a6e25  2019-05-21T15:16:25.696Z         0.0
4   ff12891e-5786-438b-891c-abd4244723b4   2019-05-22T07:51:17.49Z         0.0
5   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T08:11:13.948Z      1196.0
6   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T11:52:59.897Z     13306.0
7   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T11:53:03.406Z         4.0
8   ff12891e-5786-438b-891c-abd4244723b4  2019-05-22T11:53:03.481Z         0.0
9   fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T14:23:08.147Z         0.0
10  fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T14:29:18.228Z       370.0
11  fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T15:17:09.831Z      2872.0
12  fe55bb22-fe5b-4b12-8aaf-d5f0320ac7fa  2019-05-21T15:17:09.834Z         0.0
13  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T14:02:19.072Z         0.0
14  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T14:02:34.867Z        16.0
15  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T14:12:28.877Z       594.0
16  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T15:19:19.567Z      4011.0
17  fd3554cd-b83d-49af-a8e6-7bf41c741cd0  2019-05-21T15:19:19.582Z         0.0
18  f89c2e3e-06dc-467b-813b-dc92f2692f63  2019-05-21T09:58:02.185Z         0.0
19  f89c2e3e-06dc-467b-813b-dc92f2692f63  2019-05-21T10:07:24.044Z       562.0

或者如果需要将时间戳替换为日期时间，请使用@yatu答案。

如何使用groupby执行引用数据框中数据子集上一行的功能

2 个答案: