Question

我有一个包含数百万个组的数据框。我正在尝试为每个组在每个组的顶部添加3个月的日期（月末日期）。因此，如果组的第一次观察是2019年12月，我想在该观察之前填充3行，其日期为2019年9月至2019年11月。我还想用相关的组ID填充组列，而其他列可以保留为空值。

如果可能的话，希望避免循环，因为这是一个非常大的数据集

这是我之前的DataFrame：

import pandas as pd

before = pd.DataFrame({'Group':[1,1,1,1,1,2,2,2,2,2],
    'Date':['31/10/2018','30/11/2018','31/12/2018','31/01/2019','28/02/2019','30/03/2001','30/04/2001','31/05/2001','30/06/2001','31/07/2001'],
    'value':[1.1,1.7,1.9,2.3,1.5,2.8,2,2,2,2]})

这是我的DataFrame之后

import pandas as pd

after = pd.DataFrame({'Group':[1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2],
'Date':['31/07/2018','31/08/2018','30/09/2018','31/10/2018','30/11/2018','31/12/2018','31/01/2019','28/02/2019','31/12/2000','31/01/2001','28/02/2001','30/03/2001','30/04/2001','31/05/2001','30/06/2001','31/07/2001'],
'value':[np.nan,np.nan,np.nan,1.1,1.7,1.9,2.3,1.5,np.nan,np.nan,np.nan,2.8,2,2,2,2]})

Answer 1

因为如果多个组的解决方案不能很快地分开处理每个组，则想法是将Group的第一行乘以DataFrame.drop_duplicates，将月份排成offsets.MonthOffset，合并在一起并添加所有缺少的项日期之间：

before['Date'] = pd.to_datetime(before['Date'], dayfirst=True)

df1 = before.drop_duplicates('Group')

#first and last shifted months - by 1 and by 3 months
df11 = df1[['Group','Date']].assign(Date = lambda x: x['Date'] - pd.offsets.MonthOffset(3))
df12 = df1[['Group','Date']].assign(Date = lambda x: x['Date'] - pd.offsets.MonthOffset(1))

df = (pd.concat([df11, df12], sort=False, ignore_index=True)
       .set_index('Date')
       .groupby('Group')
       .resample('m')
       .size()
       .reset_index(name='value')
       .assign(value = np.nan))
print (df)
   Group       Date  value
0      1 2018-07-31    NaN
1      1 2018-08-31    NaN
2      1 2018-09-30    NaN
3      2 2000-12-31    NaN
4      2 2001-01-31    NaN
5      2 2001-02-28    NaN

最后添加到原始文件和排序中：

df = pd.concat([before, df], ignore_index=True).sort_values(['Group','Date'])

print (df)
    Group       Date  value
10      1 2018-07-31    NaN
11      1 2018-08-31    NaN
12      1 2018-09-30    NaN
0       1 2018-10-31    1.1
1       1 2018-11-30    1.7
2       1 2018-12-31    1.9
3       1 2019-01-31    2.3
4       1 2019-02-28    1.5
13      2 2000-12-31    NaN
14      2 2001-01-31    NaN
15      2 2001-02-28    NaN
5       2 2001-03-30    2.8
6       2 2001-04-30    2.0
7       2 2001-05-31    2.0
8       2 2001-06-30    2.0
9       2 2001-07-31    2.0

如果只有新的几个月，则可以省略groupby部分：

before['Date'] = pd.to_datetime(before['Date'], dayfirst=True)

df1 = before.drop_duplicates('Group')

df11 = df1[['Group','Date']].assign(Date = lambda x: x['Date'] - pd.offsets.MonthOffset(3))
df12 = df1[['Group','Date']].assign(Date = lambda x: x['Date'] - pd.offsets.MonthOffset(2))
df13 = df1[['Group','Date']].assign(Date = lambda x: x['Date'] - pd.offsets.MonthOffset(1))

df = (pd.concat([df11, df12, df13, before], ignore_index=True, sort=False)
        .sort_values(['Group','Date']))

print (df)
    Group       Date  value
0       1 2018-07-31    NaN
2       1 2018-08-31    NaN
4       1 2018-09-30    NaN
6       1 2018-10-31    1.1
7       1 2018-11-30    1.7
8       1 2018-12-31    1.9
9       1 2019-01-31    2.3
10      1 2019-02-28    1.5
1       2 2000-12-30    NaN
3       2 2001-01-30    NaN
5       2 2001-02-28    NaN
11      2 2001-03-30    2.8
12      2 2001-04-30    2.0
13      2 2001-05-31    2.0
14      2 2001-06-30    2.0
15      2 2001-07-31    2.0

在熊猫中每个组的开始处插入多个日期

1 个答案: