Question

我有两个分别称为a和b的数据框。 a的所有列应与DataFrame keyB的列b匹配。我定义了一个match函数来实现它，但是代码的速度很慢，因为 a和b的DataFrame实际上具有较大的形状。所以现在我想提高匹配两个DataFrame的速度。

import pandas as pd
import time

start=time.time()
a=pd.DataFrame({'key1':[1,5,1],'key2':[1,2,11]})
b=pd.DataFrame({'keyB':[1,2,3,4,5],'other':['q','q','w','w','r']})

def match(num,a,b,col):
    aFeat=a.iloc[num:num+1]
    bFeat=b[b['keyB'].isin([a[col].loc[num]])]
    aFeat.reset_index(drop=True,inplace=True)
    bFeat.reset_index(drop=True,inplace=True)
    new=pd.concat([aFeat,bFeat],axis=1)
    return new

newb=pd.DataFrame({})
for col in ['key1','key2']:
    newa=pd.DataFrame({})
    for num in range(len(a)):
        newa=pd.concat([newa,match(num,a,b,col)],axis=0)
    newa.reset_index(drop=True,inplace=True)
    del newa[col]
    newb.reset_index(drop=True,inplace=True)
    newb=pd.concat([newb,newa],axis=1)
    newb = newb.rename(columns={'keyB': 'keyB_'+col, 'other': 'other_'+col})

print(newb)
end=time.time()
print('time:',end-start)

Input:

a    key1  key2
0     1     1
1     5     2
2     1    11

b    keyB other
0     1     q
1     2     q
2     3     w
3     4     w
4     5     r

Output:

   key2  keyB_key1 other_key1  key1  keyB_key2 other_key2
0     1          1          q     1        1.0          q
1     2          5          r     5        2.0          q
2    11          1          q     1        NaN        NaN

Used time:

time: 0.015628576278686523

希望能为提高代码性能提供建议。

Answer 1

您可以将Series创建的b和s = b.set_index('keyB')['other'] print (s) keyB 1 q 2 q 3 w 4 w 5 r Name: other, dtype: object dfs = [] for col in ['key1','key2']: dfs.append(a[col]) val = a[col].map(s).rename('other_' + col) dfs.append(pd.Series(np.where(val.notnull(), a[col], np.nan), name='keyB_' + col)) dfs.append(val) df = pd.concat(dfs, axis=1) print (df) key1 keyB_key1 other_key1 key2 keyB_key2 other_key2 0 1 1.0 q 1 1.0 q 1 5 5.0 r 2 2.0 q 2 1 1.0 q 11 NaN NaN循环使用，以将每个系列附加到列表并将最后map附加在一起：

dfs = [b.merge(a[[col]], left_on='keyB', right_on=col)
        .rename(columns={'keyB':'keyB_'+col,'other':'other_'+col}) for col in ['key1','key2']]
df = pd.concat(dfs, axis=1)
print (df)
   keyB_key1 other_key1  key1  keyB_key2 other_key2  key2
0          1          q     1        1.0          q   1.0
1          1          q     1        2.0          q   2.0
2          5          r     5        NaN        NaN   NaN

列表理解为concat且列表为merge的另一种解决方案：

href

python：如何提高合并两个DataFrames的速度？

1 个答案: