Question

我有这个数据框：

df = pd.DataFrame({'a' : np.random.randn(9),
             'b' : ['foo', 'bar', 'blah'] * 3,
             'c' : np.random.randn(9)})

此功能：

def my_test2(row, x):
    if x == 'foo':
        blah = 10
    if x == 'bar':
        blah = 20
    if x == 'blah':
        blah = 30
    return (row['a'] % row['c']) + blah

然后我创建了3个这样的新列：

df['Value_foo'] = df.apply(my_test2, axis=1, x='foo')
df['Value_bar'] = df.apply(my_test2, axis=1, x='bar')
df['Value_blah'] = df.apply(my_test2, axis=1, x='blah')

它运行正常，但是当我使my_test2更复杂并将df扩展到几千行时它很慢 - 就是上面我所听到的描述为＆＃34;矢量化＆＃34;？我可以轻松加快速度吗？

Answer 1

正如Andrew，Ami Tavory和Sohier Dane已经在评论中提到的那样，有两个＆＃34;慢＆＃34;解决方案中的内容：

.apply()通常很慢，因为它在引擎盖下循环。
.apply(..., axis=1) 非常缓慢（甚至与.apply(..., axis=0)相比），因为它以行为基础进行循环

这是一种矢量化方法：

In [74]: d = {
   ....:   'foo': 10,
   ....:   'bar': 20,
   ....:   'blah': 30
   ....: }

In [75]: d
Out[75]: {'bar': 20, 'blah': 30, 'foo': 10}

In [76]: for k,v in d.items():
   ....:         df['Value_{}'.format(k)] = df.a % df.c + v
   ....:

In [77]: df
Out[77]:
          a     b         c  Value_bar  Value_blah  Value_foo
0 -0.747164   foo  0.438713  20.130262   30.130262  10.130262
1 -0.185182   bar  0.047253  20.003828   30.003828  10.003828
2  1.622818  blah -0.730215  19.432174   29.432174   9.432174
3  0.117658   foo  1.530249  20.117658   30.117658  10.117658
4  2.536363   bar -0.100726  19.917499   29.917499   9.917499
5  1.128002  blah  0.350663  20.076014   30.076014  10.076014
6  0.059516   foo  0.638910  20.059516   30.059516  10.059516
7 -1.184688   bar  0.073781  20.069590   30.069590  10.069590
8  1.440576  blah -2.231575  19.209001   29.209001   9.209001

针对90K行的时间DF：

In [80]: big = pd.concat([df] * 10**4, ignore_index=True)

In [81]: big.shape
Out[81]: (90000, 3)

In [82]: %%timeit
   ....: big['Value_foo'] = big.apply(my_test2, axis=1, x='foo')
   ....: big['Value_bar'] = big.apply(my_test2, axis=1, x='bar')
   ....: big['Value_blah'] = big.apply(my_test2, axis=1, x='blah')
   ....:
1 loop, best of 3: 10.5 s per loop

In [83]: big = pd.concat([df] * 10**4, ignore_index=True)

In [84]: big.shape
Out[84]: (90000, 3)

In [85]: %%timeit
   ....: for k,v in d.items():
   ....:     big['Value_{}'.format(k)] = big.a % big.c + v
   ....:
100 loops, best of 3: 7.24 ms per loop

结论：矢量化方法快1450倍......

是这种方法＆＃34;矢量化＆＃34; - 用于中等数据集，它相对较慢

1 个答案: