Question

所以我想优化我的代码，它从df中的列在数据帧（具有〜10 ^ 6行）中创建一个新列。我知道，以axis = 1进行应用是不好的。这是模拟的代码

这是我探索的一些链接。

https://engineering.upside.com/a-beginners-guide-to-optimizing-pandas-code-for-speed-c09ef2c6a4d6 https://maxpowerwastaken.github.io/blog/pandas-dont-apply-_-vectorize/ https://tomaugspurger.github.io/modern-4-performance.html

https://pastebin.com/1biaYCaW

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randint(0,500,size=(10000, 2)), 
                                    columns=list('AB'))
df['C'] = pd.Series(np.random.randint(5,10,size=(10000)))
df['D'] = pd.Series(np.random.randint(1,5,size=(10000)))
df_map = pd.concat([pd.Series(df['D'].unique(), name='D'),
                    pd.Series(np.random.randint(5, 10,size=(4)), name='Map')],
                  axis=1)

def manipulate(b, c, d):
    if b == c:
        return 20
    elif c == df_map[df_map['D'] == d]['Map'].values[0]:
        return 50
    else:
        return 30

__vec_manipulate = np.vectorize(manipulate)
__vec_manipulate(df['B'].values,
                 df['C'].values,
                 df['D'].values)

具有10 ^ 6行（原始代码）大约需要16分钟。想要改善这一点。

Answer 1

IIUC，这是map和np.select的问题：

# this will replace the
# c == df_map[df_map['D'] == d]['Map'].values[0]
D_map = df['D'].map(df_map.set_index('D').Map)

# your return column
np.select((df.B.eq(df.C), df.C.eq(D_map)), 
          (20,50), 
          default=30)

优化数据帧操作

1 个答案: