Question

我有一个表格的数据框（9k条目）将所有其他不相关的列分组到other_cols：

print(df_test[0:4])
   index  TF-Objektnummer Date                           other_cols
0      0          4259619 1970-01-01 13:45:41.000014557  xx
1      1          4186279 2014-10-20 06:42:23.000056098  yy
2      2          4185787 2014-10-16 06:18:56.000067086  zz
3      3          4259599 1970-01-01 13:03:59.000083584  kk

日期是具有随机时间（来自fillna）或安装日期时间的unix-epoch。

我想要获得的是一个多索引数据框，其中0级索引是TF-Objektnummer，1级索引是按以下方式定义的日期时间范围（inter =：给定ind的inter = df_test.loc [ind]）：

minff=pd.to_datetime('2001-01-07 00:00:00')
maxMD=pd.to_datetime('2017-08-31 08:33:34.000057100')
if inter["Date"]< minff:
    start=minff
else:
    start=inter["Date"]
dt_index=pd.date_range(start=start,end=maxMD)

换句话说，对于每个TF-Objektnummer，我想创建一个在level1上由Date到maxMD范围内的所有日期编制索引的行，此外我想添加level1索引的年，日，季等列。它看起来像这样：

                                       index TF-Objektnummer Date                          other_cols Year Month Day DoW Season Hour
TF-objNr Date                                                                  
4259619  2001-01-07 00:00:00.000000000     0         4259619 1970-01-01 13:45:41.000014557 xx         2001     1   7   6      0    0  
         2001-01-08 00:00:00.000000000     0         4259619 1970-01-01 13:45:41.000014557 xx         2001     1   8   0      0    0
         ...
         ...
         2017-08-30 00:00:00.000000000     0         4259619 1970-01-01 13:45:41.000014557 xx         2017     8  30   2      2    0
         2017-08-31 00:00:00.000000000     0         4259619 1970-01-01 13:45:41.000014557 xx         2017     8  31   3      2    0
...
...
4185787  2014-10-16 06:18:56.000067086     2         4185787 2014-10-16 06:18:56.000067086 zz         2014    10  16   3      3     6
         2014-10-17 06:18:56.000067086     2         4185787 2014-10-16 06:18:56.000067086 zz         2014    10  17   4      3     6 
         ...
         ...
         2017-08-31 06:18:56.000067086     2         4185787 2014-10-16 06:18:56.000067086 zz         2017     8  31   3      2     6
...

以下代码执行此操作，但是等待缓慢（如100个不同的level0索引为6h，我必须达到9k）（cols只是这里不相关的other_cols列表，但需要）

kk=0
for ind in df_test.index:
    print("#############",kk-1)
    inter=df_test.loc[ind]

    if inter["Date"]< minff:
        start=minff
    else:
        start=inter["Date"]
    dt_index=pd.date_range(start=start,end=maxMD)
    ind0=inter['TF-Objektnummer']
    cc=0
    for indd in dt_index:
        inter2=pd.DataFrame(index= pd.MultiIndex.from_tuples([(ind0,indd)], names=['TF-objNr','Date']),columns=cols+["Year","Month","Day","DoW","Season","Hour"])
        if cc%523==12:
            print(cc)
        inter2.loc[ind0,indd][cols]=inter[cols]
        inter2.loc[ind0,indd]["Year"]=indd.year
        mm=indd.month
        inter2.loc[ind0,indd]["Month"]=mm
        inter2.loc[ind0,indd]["Day"]=indd.day        
        inter2.loc[ind0,indd]["DoW"]=indd.dayofweek        
        inter2.loc[ind0,indd]["Hour"]=indd.round("h").hour
        if mm in [12,1,2]:
            inter2.loc[ind0,pd.to_datetime(indd)]["Season"]=0
        elif mm in [3,4,5]:
            inter2.loc[ind0,pd.to_datetime(indd)]["Season"]=1
        elif mm in [6,7,8]:
            inter2.loc[ind0,pd.to_datetime(indd)]["Season"]=2
        else:
            inter2.loc[ind0,pd.to_datetime(indd)]["Season"]=3

        cc+=1
        if kk==0:
            df_timeser=inter2.copy()
            kk+=1
        else:
            df_timeser=df_timeser.append(inter2)
    if kk%500==100:
        df_timeser.to_csv("/myDir/df_timeser"+str(kk)+"_"+str(ind0)+".csv",index=True)
    kk+=1

如果有人想知道，我想更好地制作一个预测模型并用天气预报来丰富它，所以我需要每天都有一个条目（TF-Objektnummer给出的位置）

Answer 1

我认为你可以使用：

from  itertools import product

minff=pd.to_datetime('2001-01-07 00:00:00')
maxMD=pd.to_datetime('2017-08-31 08:33:34.000057100')

#create tuples with replace Dates by condition
tup = list(zip(df["TF-Objektnummer"].tolist(), 
               df["Date"].mask(df["Date"] < minff, minff).tolist()))

print (tup)
[(4259619, Timestamp('2001-01-07 00:00:00')), 
 (4186279, Timestamp('2014-10-20 06:42:23.000056098')), 
 (4185787, Timestamp('2014-10-16 06:18:56.000067086')), 
 (4259599, Timestamp('2001-01-07 00:00:00'))]

#create product of date ranges and flatten output
tup1 =  [i for a, b in tup for i in list(product([a], pd.date_range(start=b,end=maxMD)))]
#final MultiIndex
mux = pd.MultiIndex.from_tuples(tup1, names=['TF-objNr','Date'])

#reindex by MultiIndex
df = df.set_index('TF-Objektnummer').reindex(mux, level=0)

#add new columns
idd = df.index.get_level_values(1)
df['Year'] = idd.year
df['Month'] = idd.month
df['Day'] = idd.day
df['DoW'] = idd.dayofweek
df['Hour'] = idd.round("h").hour
df["Season"] = (df['Month'] % 12 + 3) // 3 - 1

print (df.head())
                     index                          Date other_cols  Year  \
TF-objNr Date                                                               
4259619  2001-01-07      0 1970-01-01 13:45:41.000014557         xx  2001   
         2001-01-08      0 1970-01-01 13:45:41.000014557         xx  2001   
         2001-01-09      0 1970-01-01 13:45:41.000014557         xx  2001   
         2001-01-10      0 1970-01-01 13:45:41.000014557         xx  2001   
         2001-01-11      0 1970-01-01 13:45:41.000014557         xx  2001   

                     Month  Day  DoW  Hour  Season  
TF-objNr Date                                       
4259619  2001-01-07      1    7    6     0       0  
         2001-01-08      1    8    0     0       0  
         2001-01-09      1    9    1     0       0  
         2001-01-10      1   10    2     0       0  
         2001-01-11      1   11    3     0       0

多索引数据框创建加速

1 个答案: