Question

我正在使用numpy.array（）函数从列表中创建numpy.float64 ndarrays。

我注意到，当列表包含None或列表列表时，这非常慢。

下面是一些有时间的例子。有明显的解决方法，但为什么这么慢？

无名单的例子：

### Very slow to call array() with list of None
In [3]: %timeit numpy.array([None]*100000, dtype=numpy.float64)
1 loops, best of 3: 240 ms per loop

### Problem doesn't exist with array of zeroes
In [4]: %timeit numpy.array([0.0]*100000, dtype=numpy.float64)
100 loops, best of 3: 9.94 ms per loop

### Also fast if we use dtype=object and convert to float64
In [5]: %timeit numpy.array([None]*100000, dtype=numpy.object).astype(numpy.float64)
100 loops, best of 3: 4.92 ms per loop

### Also fast if we use fromiter() insead of array()
In [6]: %timeit numpy.fromiter([None]*100000, dtype=numpy.float64)
100 loops, best of 3: 3.29 ms per loop

列表清单的示例：

### Very slow to create column matrix
In [7]: %timeit numpy.array([[0.0]]*100000, dtype=numpy.float64)
1 loops, best of 3: 353 ms per loop

### No problem to create column vector and reshape
In [8]: %timeit numpy.array([0.0]*100000, dtype=numpy.float64).reshape((-1,1))
100 loops, best of 3: 10 ms per loop

### Can use itertools to flatten input lists
In [9]: %timeit numpy.fromiter(itertools.chain.from_iterable([[0.0]]*100000),dtype=numpy.float64).reshape((-1,1))
100 loops, best of 3: 9.65 ms per loop

Answer 1

我已将此报告为一个numpy问题。报告和补丁文件位于：

https://github.com/numpy/numpy/issues/3392

修补后：

# was 240 ms, best alternate version was 3.29
In [5]: %timeit numpy.array([None]*100000)
100 loops, best of 3: 7.49 ms per loop

# was 353 ms, best alternate version was 9.65
In [6]: %timeit numpy.array([[0.0]]*100000)
10 loops, best of 3: 23.7 ms per loop

Answer 2

我的猜测是，转换列表的代码只会在所有内容上调用float。如果参数定义__float__，我们称之为，否则我们将其视为一个字符串（在None上抛出异常，我们捕获它并放入np.nan）。异常处理应该相对较慢。

时间似乎验证了这个假设：

import numpy as np
%timeit [None] * 100000
> 1000 loops, best of 3: 1.04 ms per loop

%timeit np.array([0.0] * 100000)
> 10 loops, best of 3: 21.3 ms per loop
%timeit [i.__float__() for i in [0.0] * 100000]
> 10 loops, best of 3: 32 ms per loop


def flt(d):
    try:
        return float(d)
    except:
        return np.nan

%timeit np.array([None] * 100000, dtype=np.float64)
> 1 loops, best of 3: 477 ms per loop    
%timeit [flt(d) for d in [None] * 100000]
> 1 loops, best of 3: 328 ms per loop

添加另一个案例只是为了明白我要去哪里。如果有明确的无效检查，则上面的速度不会很慢：

def flt2(d):                              
    if d is None:
        return np.nan
    try:
        return float(d)
    except:
        return np.nan

%timeit [flt2(d) for d in [None] * 100000]
> 10 loops, best of 3: 45 ms per loop

为什么numpy.array（）有时很慢？

2 个答案: