Question

在我的Anaconda Python发行版上，复制一个16 GB或更大的Numpy数组（无论dtype如何）都会将副本的所有元素设置为0：

>>> np.arange(2 ** 31 - 1).copy()  # works fine
array([         0,          1,          2, ..., 2147483644, 2147483645,
       2147483646])
>>> np.arange(2 ** 31).copy()  # wait, what?!
array([0, 0, 0, ..., 0, 0, 0])
>>> np.arange(2 ** 32 - 1, dtype=np.float32).copy()
array([  0.00000000e+00,   1.00000000e+00,   2.00000000e+00, ...,
         4.29496730e+09,   4.29496730e+09,   4.29496730e+09], dtype=float32)
>>> np.arange(2 ** 32, dtype=np.float32).copy()
array([ 0.,  0.,  0., ...,  0.,  0.,  0.], dtype=float32)

此分发的np.__config__.show()：

blas_opt_info:
    library_dirs = ['/users/username/.anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/users/username/.anaconda3/include']
    libraries = ['mkl_rt', 'pthread']
lapack_opt_info:
    library_dirs = ['/users/username/.anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/users/username/.anaconda3/include']
    libraries = ['mkl_rt', 'pthread']
mkl_info:
    library_dirs = ['/users/username/.anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/users/username/.anaconda3/include']
    libraries = ['mkl_rt', 'pthread']
openblas_lapack_info:
  NOT AVAILABLE
lapack_mkl_info:
    library_dirs = ['/users/username/.anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/users/username/.anaconda3/include']
    libraries = ['mkl_rt', 'pthread']
blas_mkl_info:
    library_dirs = ['/users/username/.anaconda3/lib']
    define_macros = [('SCIPY_MKL_H', None), ('HAVE_CBLAS', None)]
    include_dirs = ['/users/username/.anaconda3/include']
    libraries = ['mkl_rt', 'pthread']

为了比较，这里是我的系统Python发行版的np.__config__.show()，它没有这个问题：

blas_opt_info:
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
    language = c
    library_dirs = ['/usr/local/lib']
openblas_lapack_info:
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
    language = c
    library_dirs = ['/usr/local/lib']
openblas_info:
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
    language = c
    library_dirs = ['/usr/local/lib']
lapack_opt_info:
    define_macros = [('HAVE_CBLAS', None)]
    libraries = ['openblas', 'openblas']
    language = c
    library_dirs = ['/usr/local/lib']
blas_mkl_info:
  NOT AVAILABLE

我想知道MKL加速是否是问题。我已经在Python 2和3上重现了这个bug。

Answer 1

这只是猜测。我目前没有任何证据支持以下声明，但我的猜测是这是一个简单的溢出问题：

>>> np.arange(2 ** 31 - 1).size
2147483647

恰好是最大的int32值：

>>> np.iinfo(np.int32)
iinfo(min=-2147483648, max=2147483647, dtype=int32)

因此，当您实际拥有大小为2147483648（2**31）的数组并使用int32时，这将溢出并给出实际的负值。然后在numpy.ndarray.copy方法中可能存在类似的内容：

for (i = 0 ; i < size ; i ++) {
    newarray[i] = oldarray[i]
}

但是考虑到现在大小为负，循环将不会执行，因为0 > -2147483648。

新数组实际上用零初始化是很奇怪的，因为在复制数组之前实际放置零是没有意义的（但它可能类似于in this question）。

再说一遍：这只是在猜测，但它会与行为相匹配。

为什么复制一个＆gt; = 16 GB的Numpy数组会将其所有元素设置为0？

1 个答案: