Question

我有一个遍历一维数组的函数，并检查这些值是否超过阈值以创建掩码。非常快。但是我如何使用它来遍历在不同列上具有不同阈值的多个列。到目前为止，对于尺寸为18531的一维阵列，我的方法花费了12 µs的时间。如果我遍历两列，则尝试了不同的功能，但只有一个阈值。如何使用多个阈值执行此操作？此外，我调查了使用numpy floats 16或numpy floats32的速度要慢得多。为什么会这样？

import numpy as np
import numba
import pandas as pd
#######Column approach
@numba.jit
def compute_expressionCol_Numba(col,threshold):
    n=len(col)
    result = np.empty(n,dtype='bool')
    for i in range(n):
        if col[i] < threshold:
            result[i]=1
        else:
            result[i]=0
    return result

def compute_expressionCol(col,threshold):
    result = compute_expressionCol_Numba(col.values,threshold)
    return result

##### Multiple column approach

def compute_expressionDF(df,threshold):
    for i in df:
        result = compute_expressionCol_Numba(df[i].values,threshold)
    return result

def make_mask(df, threshold):
    result = np.where(df < threshold, 1 , 0)
    return result

def lt(df, thresh):
    return (df.values<thresh).view('i1')

import numexpr as ne

def lt_numexpr(df, thresh):
    return ne.evaluate('a<thresh',{'a':df.values})

一些timeit测试：

for i in [np.float16,np.float32,np.float64]:
    print(i)
    randomDF = pd.DataFrame(np.random.rand(19000,2).astype(i),columns=['col1','col2'])
    thresh = 50
    %timeit compute_expressionCol(randomDF['col1'],50)
    %timeit compute_expressionCol(randomDF['col2'],50)
    %timeit for i in randomDF[['col1','col2']]: compute_expressionCol(randomDF[i],50)
    %timeit (randomDF[['col1','col2']].values < 50).astype(int)
    %timeit (randomDF.values < 50).astype(int)
    %timeit make_mask(randomDF[['col1','col2']],50)
    %timeit randomDF[['col1','col2']]<50
    %timeit randomDF['col1']<50
    %timeit pd.eval('randomDF[["col1","col2"]]<50')
    %timeit lt(randomDF, thresh=50)
    %timeit lt_numexpr(randomDF, thresh=50)

结果：

<class 'numpy.float16'>
40 ms ± 1.05 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
40.6 ms ± 1.28 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
82.8 ms ± 1.37 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
1.39 ms ± 40.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
416 µs ± 14.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
2.88 ms ± 31.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.95 ms ± 97.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
400 µs ± 8.08 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
31.2 ms ± 1.06 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
413 µs ± 22.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
598 µs ± 22.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
<class 'numpy.float32'>
30.3 µs ± 2.31 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
35.2 µs ± 2.81 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
1.01 ms ± 67 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
973 µs ± 93.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
37.5 µs ± 4.76 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
3.11 ms ± 544 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.56 ms ± 129 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
224 µs ± 10.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
32.4 ms ± 3.75 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
70.8 µs ± 2.19 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
555 µs ± 18.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
<class 'numpy.float64'>
26.5 µs ± 2.09 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
27.2 µs ± 836 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
1.09 ms ± 62.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
1.01 ms ± 38.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
43.1 µs ± 2.19 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
2.78 ms ± 118 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
2.54 ms ± 35.5 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
249 µs ± 10.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
30.4 ms ± 1.07 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
71.8 µs ± 3.05 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
558 µs ± 21 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Answer 1

我认为np.where可能就是您想要的。您可以为它提供一个数据框或一系列数据

import numpy as np

def make_mask(df, threshold):
    result = np.where(df < threshold, 1 , 0)
    return result

Python：Numba遍历多维数组

1 个答案: