我有2个矩阵100kx200和200x100k 如果它们是小矩阵我就会使用numpy dot product
sum(a.dot(b), axis = 0)
然而矩阵太大了,我也不能使用循环是否有一种聪明的方法可以做到这一点?
答案 0 :(得分:18)
可能的优化是
SELECT GREATEST(column1, column2) as greatest from Table
计算>>> numpy.sum(a @ b, axis=0)
array([ 1.83633615, 18.71643672, 15.26981078, -46.33670382, 13.30276476])
>>> numpy.sum(a, axis=0) @ b
array([ 1.83633615, 18.71643672, 15.26981078, -46.33670382, 13.30276476])
需要10k×200×10k的运算,而首先对行进行求和会将乘法运算减少到1×200×10k,从而实现10kx的改进。
这主要是由于认识到了
a @ b
与其他轴类似。
numpy.sum(x, axis=0) == [1, 1, ..., 1] @ x
=> numpy.sum(a @ b, axis=0) == [1, 1, ..., 1] @ (a @ b)
== ([1, 1, ..., 1] @ a) @ b
== numpy.sum(a, axis=0) @ b
(注意:x @ y
相当于{3.5}上2D matrixes and 1D vectors numpy 1.10.0+的>>> numpy.sum(a @ b, axis=1)
array([ 2.8794171 , 9.12128399, 14.52009991, -8.70177811, -15.0303783 ])
>>> a @ numpy.sum(b, axis=1)
array([ 2.8794171 , 9.12128399, 14.52009991, -8.70177811, -15.0303783 ])
{/ 3}}
x.dot(y)
<强>插图:强>
$ INITIALIZATION='import numpy;numpy.random.seed(0);a=numpy.random.randn(1000,200);b=numpy.random.rand(200,1000)'
$ python3 -m timeit -s "$INITIALIZATION" 'numpy.einsum("ij,jk->k", a, b)'
10 loops, best of 3: 87.2 msec per loop
$ python3 -m timeit -s "$INITIALIZATION" 'numpy.sum(a@b, axis=0)'
100 loops, best of 3: 12.8 msec per loop
$ python3 -m timeit -s "$INITIALIZATION" 'numpy.sum(a, axis=0)@b'
1000 loops, best of 3: 300 usec per loop
现在,如果我们只是In [235]: a = np.random.rand(3,3)
array([[ 0.465, 0.758, 0.641],
[ 0.897, 0.673, 0.742],
[ 0.763, 0.274, 0.485]])
In [237]: b = np.random.rand(3,2)
array([[ 0.303, 0.378],
[ 0.039, 0.095],
[ 0.192, 0.668]])
,我们需要 18个乘法和6个加法运算。另一方面,如果我们a @ b
,我们只需要 6乘法和2个加法运算。由于我们在np.sum(a, axis=0) @ b
中有3行,因此提高了3倍。至于OP的情况,这应该比简单a
计算提高10k倍,因为他在a @ b
中有10k行。
答案 1 :(得分:5)
有两个sum-reductions
正在发生 - 一个来自marix-multiplelication np.dot
,另一个来自显式sum
。
我们可以使用np.einsum
一次完成这两项操作,就像这样 -
np.einsum('ij,jk->k',a,b)
示例运行 -
In [27]: a = np.random.rand(3,4)
In [28]: b = np.random.rand(4,3)
In [29]: np.sum(a.dot(b), axis = 0)
Out[29]: array([ 2.70084316, 3.07448582, 3.28690401])
In [30]: np.einsum('ij,jk->k',a,b)
Out[30]: array([ 2.70084316, 3.07448582, 3.28690401])
运行时测试 -
In [45]: a = np.random.rand(1000,200)
In [46]: b = np.random.rand(200,1000)
In [47]: %timeit np.sum(a.dot(b), axis = 0)
100 loops, best of 3: 5.5 ms per loop
In [48]: %timeit np.einsum('ij,jk->k',a,b)
10 loops, best of 3: 71.8 ms per loop
可悲的是,np.einsum
看起来好像我们没有做得更好。
要更改为np.sum(a.dot(b), axis = 1)
,只需在那里交换输出字符串表示法 - np.einsum('ij,jk->i',a,b)
,就像这样 -
In [42]: np.sum(a.dot(b), axis = 1)
Out[42]: array([ 3.97805141, 3.2249661 , 1.85921549])
In [43]: np.einsum('ij,jk->i',a,b)
Out[43]: array([ 3.97805141, 3.2249661 , 1.85921549])
答案 2 :(得分:2)
使用我在Divakar的回答中添加的想法进行了一些快速测试:
In [162]: a = np.random.rand(1000,200)
In [163]: b = np.random.rand(200,1000)
In [174]: timeit c1=np.sum(a.dot(b), axis=0)
10 loops, best of 3: 27.7 ms per loop
In [175]: timeit c2=np.sum(a,axis=0).dot(b)
1000 loops, best of 3: 432 µs per loop
In [176]: timeit c3=np.einsum('ij,jk->k',a,b)
10 loops, best of 3: 170 ms per loop
In [177]: timeit c4=np.einsum('j,jk->k', np.einsum('ij->j', a), b)
1000 loops, best of 3: 353 µs per loop
In [178]: timeit np.einsum('ij->j', a) @b
1000 loops, best of 3: 304 µs per loop
einsum
实际上比np.sum
快!
In [180]: timeit np.einsum('ij->j', a)
1000 loops, best of 3: 173 µs per loop
In [181]: timeit np.sum(a,0)
1000 loops, best of 3: 312 µs per loop
对于较大的数组,einsum
优势会降低
In [183]: a = np.random.rand(100000,200)
In [184]: b = np.random.rand(200,100000)
In [185]: timeit np.einsum('ij->j', a) @b
10 loops, best of 3: 51.5 ms per loop
In [186]: timeit c2=np.sum(a,axis=0).dot(b)
10 loops, best of 3: 59.5 ms per loop