Question

我有3D数据（x，y，z）的随机样本，其中x和y是空间维度，z是其在平面上的位置的函数：z = f（x，y）。我想使用样本中的插值来在常规网格上评估此函数。

每组样本都在不同的文件中，需要单独插补（每个文件都是不同的时间点）。为了加快这些文件的处理速度，我想使用多处理（mp）模块使用mp池并行执行它们。

但是，当我尝试在mp池调用的函数中使用mp.griddata时，进程在进入griddata函数时会挂起。串行执行相同功能时没有问题。这也只发生在某些numpy / scipy构建中。当我的numpy.show_config（）如下：

blas_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
lapack_info:
    libraries = ['lapack']
    library_dirs = ['/usr/lib']
    language = f77
atlas_threads_info:
  NOT AVAILABLE
blas_opt_info:
    libraries = ['blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_blas_threads_info:
  NOT AVAILABLE
lapack_opt_info:
    libraries = ['lapack', 'blas']
    library_dirs = ['/usr/lib']
    language = f77
    define_macros = [('NO_ATLAS_INFO', 1)]
atlas_info:
  NOT AVAILABLE
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
atlas_blas_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

一切正常。

但是当配置为：

lapack_info:
    libraries = ['openblas']
    library_dirs = ['/u/ccor/local/lib']
    language = f77
atlas_threads_info:
    libraries = ['openblas']
    library_dirs = ['/u/ccor/local/lib']
    define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
    language = c
    include_dirs = ['/usr/include']
blas_opt_info:
    libraries = ['openblas']
    library_dirs = ['/u/ccor/local/lib']
    define_macros = [('ATLAS_INFO', '"\\"None\\""')]
    language = c
    include_dirs = ['/usr/include']
atlas_blas_threads_info:
    libraries = ['openblas']
    library_dirs = ['/u/ccor/local/lib']
    define_macros = [('ATLAS_INFO', '"\\"None\\""')]
    language = c
    include_dirs = ['/usr/include']
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['/u/ccor/local/lib']
    define_macros = [('ATLAS_WITHOUT_LAPACK', None)]
    language = f77
    include_dirs = ['/usr/include']
lapack_mkl_info:
  NOT AVAILABLE
blas_mkl_info:
  NOT AVAILABLE
mkl_info:
  NOT AVAILABLE

子进程挂起。

第二个配置是通过手动编译LibOpenBLAS和numpy来尝试优化我的BLAS库以用于具有多个核心的机器。在这两种情况下，我都使用python 2.7.3进行虚拟操作。

此外，挂起仅在griddata与'linear'或'cubic'插值一起使用时发生; '最近'工作得很好。

以下是一些试图说明和隔离问题的代码。数据是随机生成的，而不是从文件中读取，结果只是作为一个例子随意累积。

import multiprocessing as mp
import numpy
from scipy import interpolate

def interpolate_grid(it, n_points, grid_dim):

    print 'interpolating: ', it

    numpy.random.seed(it)

    points = 2*numpy.random.random((n_points,2))-0.5
    values = numpy.random.random(n_points)
    x,y = numpy.mgrid[0:1:grid_dim*1j,0:1:grid_dim*1j]
    grid_pnts = numpy.vstack([x.flatten(), y.flatten()]).T
    interp_vals = interpolate.griddata(points, values, grid_pnts, 'cubic')
    print 'done interpolating: ', it
    return interp_vals

def interpolation_test(n_runs = 10, n_points = 1000, grid_dim = 10, parallel = True):

    class Sum(object):
        def __init__(self, grid_dim):
            self.acc = numpy.zeros(grid_dim*grid_dim)

        def accumulate(self, val):
            self.acc += val

    acc = Sum(grid_dim)

    if parallel:
        pool = mp.Pool(mp.cpu_count())
        for i in xrange(n_runs):
            pool.apply_async(interpolate_grid, 
                            args = [i, n_points, grid_dim], 
                            callback = acc.accumulate)
        pool.close()
        pool.join()

    else:
        for i in xrange(n_runs):
            acc.acc += interpolate_grid(i, n_points, grid_dim)

    return acc.acc

def main():

    ser_result = interpolation_test(parallel = False)
    par_result = interpolation_test(parallel = True)

    assert (numpy.abs(par_result-ser_result) < 1e-14).all()
    print 'serial and parallel results the same'

if __name__ == '__main__':
    main()

编辑：我在scikit-learn installation page上发现了这一点：“使用OpenBLAS可以在某些scikit-learn模块中提供加速，但它不能很好地与joblib / multiprocessing一起使用，所以除非你知道，否则不建议使用它你正在做什么。“

我通过使用ATLAS而不是OpenBLAS找到了一种解决方法：

aptitude install libatlas-base-dev
ATLAS=/usr/lib
pip install numpy

它不如OpenBLAS快，但适用于多处理，这对我的应用程序更重要，并且比基本的numpy安装快得多。有谁“知道他们在做什么”关心为什么OpenBLAS和多处理“不能很好地结合在一起”？或者如何让他们玩得更好？ scikit-learn网站上的警告似乎暗示它是可能的。

为什么scipy.interpolate.griddata在与多处理和OpenBLAS一起使用时会挂起？

0 个答案: