Question

您能告诉我一种优化此代码的方法吗？由于数据集庞大，需要数十分钟才能完成...

df['sinistre'] = 0
for index_sin, row_sin in sinistre1.iterrows():
    date_surv = row_sin['DATESURV']
    quit_sin = df.loc[df['id_police'] == row_sin['id_police']]
    for index, row in quit_sin.iterrows():
        if row['DATEEFFE'] < date_surv < row['DATE_FIN']:
            df['sinistre'][index] = 1

以下是数据帧sinistre1和df的示例数据集：

>>> sinistre1
  id_police id_sinistre    DATESURV
0      p123        s123  30/05/2017
1      p123        s124  30/11/2017
2      p123        s125  29/02/2018
3      b123        s126  28/02/2018
4      b123        s127  30/05/2018
>>> df
  id_police    DATEEFFE    DATE_FIN  prime  prime2
0      p123  24/01/2017  24/02/2017      0       0
1      p123  24/11/2017  24/12/2017      0      30
2      p123  25/02/2018  25/03/2018     10      10
3      b123  24/02/2018  24/03/2018     20      20
4      b123  24/03/2018  24/04/2018     30       0

这是预期的输出（这个想法是，当sinistre1中的DATESURV在间隔DATEEFFE和DATE_FIN内时，我标记为sinistre）：< / p>

  id_police    DATEEFFE    DATE_FIN  prime  prime2  sinistre
0      p123  24/01/2017  24/02/2017      0       0         0
1      p123  24/11/2017  24/12/2017      0      30         1
2      p123  25/02/2018  25/03/2018     10      10         1
3      b123  24/02/2018  24/03/2018     20      20         1
4      b123  24/03/2018  24/04/2018     30       0         0

如果我无法避免for循环，请显示一种更好的循环速度更快的方法...预先感谢！

Answer 1

正如我在评论中所述。现在，可接受的答案和合并没有任何意义，因为我认为OP希望比较两个数据帧中的每一行，因此也需要数据帧id_sinistre中的键df。或想像以下一样使用combine_first：

df_merge = df.merge(sinistre1, on='id_police', how='left')
df_merge['DATESURV'] = pd.to_datetime(df_merge['DATESURV'])

df_merge['sinistre'] = np.where(df_merge['DATESURV'].between(df_merge['DATEEFFE'], df_merge['DATE_FIN']), 1, 0)

df_merge = df_merge.drop(['DATESURV', 'id_sinistre'], axis=1)

print(df_merge)
    DATEEFFE   DATE_FIN id_police  prime  prime2  sinistre
0 2017-01-24 2017-02-24      p123      0       0         0
1 2017-11-24 2017-12-24      p123      0      30         1
2 2018-02-25 2018-03-25      p123     10      10         1
3 2018-02-24 2018-03-24      b123     20      20         1
4 2018-03-24 2018-04-24      b123     30       0         0

Answer 2

在“ id_police”列上左合并两个数据集
编写具有业务逻辑的lambda函数并将其应用于合并的数据集

即（未测试）：

t_table = pd.merge(sinistre1, df, how='left', on='id_police')
t_table['sinistre'] = [1 if row['DATEEFFE'] < ds< row['DATE_FIN'] else
0 for row,_ in t_table.iterrows()]

Answer 3

使用：

用于转换所有日期时间列的第一个函数to_datetime
为最近的检查成员资格创建计数器列sinistre1
将DataFrame.merge与left联接一起用于数据
通过Series.between和inclusive=True进行过滤，仅获得系列sinistre1中的列s
通过用sinistre1和Series.isin进行校验来最后覆盖最终的s，并将其转换为整数，从True/False到1/0映射：

df['DATEEFFE'] = pd.to_datetime(df['DATEEFFE'])
df['DATE_FIN'] = pd.to_datetime(df['DATE_FIN'])
sinistre1['DATESURV'] = pd.to_datetime(sinistre1['DATESURV'])

df['sinistre1'] = np.arange(len(df))
df1 = df.merge(sinistre1, on='id_police', how='left')
mask = df1['DATESURV'].between(df1['DATEEFFE'], df1['DATE_FIN'], inclusive=False)
s = df1.loc[mask, 'sinistre1']
print (s)
4    1
8    2
9    3
Name: sinistre1, dtype: int32

df['sinistre1'] = df['sinistre1'].isin(s).astype(int)
#alternative
#df['sinistre1'] = np.where(df['sinistre1'].isin(s), 1, 0)
print (df)
  id_police   DATEEFFE   DATE_FIN  prime  prime2  sinistre1
0      p123 2017-01-24 2017-02-24      0       0          0
1      p123 2017-11-24 2017-12-24      0      30          1
2      p123 2018-02-25 2018-03-25     10      10          1
3      b123 2018-02-24 2018-03-24     20      20          1
4      b123 2018-03-24 2018-04-24     30       0          0

编辑：

df['DATEEFFE'] = pd.to_datetime(df['DATEEFFE'])
df['DATE_FIN'] = pd.to_datetime(df['DATE_FIN'])
sinistre1['DATESURV'] = pd.to_datetime(sinistre1['DATESURV'])

df1 = df.merge(sinistre1, on='id_police', how='left')
#filter all rows
df2 = df1[df1['DATESURV'].between(df1['DATEEFFE'], df1['DATE_FIN'], inclusive=False)]
print (df2)
  id_police   DATEEFFE   DATE_FIN  prime  prime2 id_sinistre   DATESURV
4      p123 2017-11-24 2017-12-24      0      30        s124 2017-11-30
8      p123 2018-02-25 2018-03-25     10      10        s125 2018-02-28
9      b123 2018-02-24 2018-03-24     20      20        s126 2018-02-28

#left join merge with inditator True
df = df.merge(df2.drop('DATESURV', axis=1), how='left', indicator=True)
#sinistre1 column - first rename and then compare both for 1, esle 0
df = df.rename(columns={'_merge':'sinistre1'})
df['sinistre1'] = df['sinistre1'].eq('both').astype(int)
print (df)
  id_police   DATEEFFE   DATE_FIN  prime  prime2 id_sinistre  sinistre1
0      p123 2017-01-24 2017-02-24      0       0         NaN          0
1      p123 2017-11-24 2017-12-24      0      30        s124          1
2      p123 2018-02-25 2018-03-25     10      10        s125          1
3      b123 2018-02-24 2018-03-24     20      20        s126          1
4      b123 2018-03-24 2018-04-24     30       0         NaN          0

如何避免大数据集的Pandas DataFrame中的for循环

3 个答案: