Question

我有以下python代码：

H1 = [[0.04,0.03,0.01,0.002],[0.02,0.04,0.001,0.5]]
H2 = [[0.06,0.02,0.02,0.004],[0.8,0.09,0.6,0.1]]    

D1 = [0.01,0.02,0.1,0.01]    
D2 = [0.1,0.3,0.01,0.4] 

Tp = np.sum(D1)    
Tn = np.sum(D2) 

T = []    
append2 = T.append   
E = []    
append3 = E.append   

for h1,h2 in itertools.izip(H1,H2) 
    Err = []    
    append1 = Err.append
    for v in h1:    

        L1 = [1 if i>=v else 0 for i in h1]    
        L2 = [1 if i>=v else 0 for i in h2]    

        Sp = np.dot(D1,L1)     
        Sn = np.dot(D2,L2)    

        err = min(Sp+Tn-Sn, Sn+Tp-Sp)    
        append1(err)  

    b = np.argmin(Err)    
    append2(h1[b])    
    append3(Err[b])

这只是一个示例代码。我需要运行内部for循环大约20,000次（这里只运行两次）。但是内部for循环需要花费很多时间才能使它变得不实用。在线条分析器中，它显示行Sp = np.dot(D1,L1)，Sn = np.dot(D2,L2)和b = np.argmin(Err)是最耗时的。如何减少上述代码所花费的时间。

非常感谢任何帮助。

谢谢！

Answer 1

如果使用numpy函数而不是列表，那么你可以获得相当大的速度提升。大多数numpy函数会在内部将列表转换为数组，这会给运行时增加很多开销。这是一个简单的例子：

In [16]: a = range(10)

In [17]: b = range(10)

In [18]: aa = np.array(a)

In [19]: bb = np.array(b)

In [20]: %timeit np.dot(a, b)
10000 loops, best of 3: 54 us per loop

In [21]: %timeit np.dot(aa, bb)
100000 loops, best of 3: 3.4 us per loop

在这种情况下，使用数组调用时，

numpy.dot运行速度提高了16倍。此外，当您使用numpy数组时，您将能够简化一些代码，这也有助于它更快地运行。例如，如果h1是一个数组，L1 = [1 if i>=v else 0 for i in h1]可以写成h1 > v，它返回一个数组，并且运行得更快。 Bellow我已经继续使用数组替换你的列表，这样你就可以看到它的外观。

import numpy as np

H1 = np.array([[0.04,0.03,0.01,0.002],[0.02,0.04,0.001,0.5]])
H2 = np.array([[0.06,0.02,0.02,0.004],[0.8,0.09,0.6,0.1]])

D1 = np.array([0.01,0.02,0.1,0.01])
D2 = np.array([0.1,0.3,0.01,0.4])

Tp = np.sum(D1)    
Tn = np.sum(D2) 

T = np.zeros(H1.shape[0])
E = np.zeros(H1.shape[0])

for i in range(len(H1)):
    h1 = H1[i]
    h2 = H2[i]
    Err = np.zeros(len(h1))

    for j in range(len(h1)):    
        v = h1[j]

        L1 = h1 > v
        L2 = h2 > v

        Sp = np.dot(D1, L1)     
        Sn = np.dot(D2, L2)    

        err = min(Sp+Tn-Sn, Sn+Tp-Sp)    
        Err[j] = err

    b = np.argmin(Err)
    T[i] = h1[b]
    E[i] = Err[b]

一旦你对numpy数组更加熟悉，你可能希望使用broadcasting来表达至少你的内部循环。对于某些应用程序，使用广播可以比python循环更有效。祝你好运，希望有所帮助。

Answer 2

你的列表理解中有一些悬而未决的成果：

L1 = [1 if i>=v else 0 for i in h1]
L2 = [1 if i>=v else 0 for i in h2]

以上内容可以写成：

L1 = [i>=v for i in h1]
L2 = [i>=v for i in h2]

因为布尔值是整数的子类，True和False已经是1和0，只穿着华丽的衣服。

err = min(Sp+Tn-Sn, Sn+Tp-Sp)    
append1(err)

您可以合并上述两行以避免变量分配和访问。

如果将代码放在函数中，则所有局部变量的使用速度会稍快一些。此外，您使用的任何全局函数或方法（例如min，np.dot）都可以使用默认参数转换为函数签名中的本地函数。 np.dot是一个特别慢的调用（在操作本身需要的时间之外），因为它涉及属性查找。这与您使用列表append方法进行的优化类似。

现在我想象这一切都不会真正影响性能，因为你的问题似乎真的是“我怎样才能让NumPy变得更快？” （其他人对你而言最重要）但他们可能会有一些影响，值得做。

Answer 3

您需要将数据保存在ndarray类型中。在列表上执行numpy操作时，每次都必须构造一个新数组。我修改了你的代码以运行可变次数，并发现10000次迭代也是1s。将数据类型更改为ndarrays减少了大约两倍，我认为仍然有一些改进（第一个版本有一个错误导致它执行得太快）

import itertools
import numpy as np
N = 10000
H1 = [np.array([0.04,0.03,0.01,0.002])] * N
H2 = [np.array([0.06,0.02,0.02,0.004])] * N

D1 = np.array([0.01,0.02,0.1,0.01]    )
D2 = np.array([0.1,0.3,0.01,0.4] )

Tp = np.sum(D1)    
Tn = np.sum(D2) 

T = []    
append2 = T.append   
E = []    
append3 = E.append   

for h1,h2 in itertools.izip(H1,H2):
    Err = []    
    append1 = Err.append
    for v in h1:

        #L1 = [1 if i>=v else 0 for i in h1]    
        #L2 = [1 if i>=v else 0 for i in h2]    
        L1 = h1 > v
        L2 = h2 > v
        Sp = np.dot(D1,L1)     
        Sn = np.dot(D2,L2)    

        err = min(Sp+Tn-Sn, Sn+Tp-Sp)    
        append1(err)  

    b = np.argmin(Err)    
    append2(h1[b])    
    append3(Err[b])

Answer 4

如果我在两个维度列表1中正确理解了指令np.dot()的内容，那么在我看来，以下代码应该与您的相同。
你能测试它的速度吗？

它的原则是使用索引而不是列表元素，并使用定义为函数默认值的列表的特性

H1 = [[0.04,0.03,0.01,0.002],[0.02,0.04,0.001,0.5]]
H2 = [[0.06,0.02,0.02,0.004],[0.8,0.09,0.6,0.1]]    

D1 = [0.01,0.02,0.1,0.01]    
D2 = [0.1,0.3,0.01,0.4] 

Tp = np.sum(D1)    
Tn = np.sum(D2) 

T,E = [],[]    
append2 = T.append      
append3 = E.append 

ONE,TWO = [],[]

def zoui(v, ONE=ONE,TWO=TWO,
         D1=D1,D2=D2,Tp=Tp,Tn=Tn,tu0123 = (0,1,2,3)):
    diff =  sum(D1[i] if ONE[i]>=v else 0 for i in tu0123)\
           -sum(D2[i] if TWO[i]>=v else 0 for i in tu0123)
    #or maybe
    #diff =  sum(D1[i] * ONE[i]>=v for i in tu0123)\
    #       -sum(D2[i] * TWO[i]>=v for i in tu0123)

    return min(Tn+diff,Tp-diff)

for n in xrange(len(H1)):
    ONE[:] = H1[n]
    TWO[:] = H2[n]
    Err = map(zoui,ONE)
    b = np.argmin(Err)    
    append2(ONE[b])    
    append3(Err[b])

如何减少python中的循环时间？

4 个答案: