Question

我将浮点数列表转换为带有以下字段的ctypes Structure类，然后再将它们传递给FFI函数：

FFIArray(Structure):
    _fields_ = [("data", c_void_p),
                ("len", c_size_t)]

    @classmethod
    def from_param(cls, seq):
        return seq if isinstance(seq, cls) else cls(seq)

    def __init__(self, seq, data_type = c_float):
        array_type = data_type * len(seq)
        raw_seq = array_type(*seq)
        self.data = cast(raw_seq, c_void_p)
        self.len = len(seq)

返回的FFIArray对象（总是与输入长度相同，输入值本身不被FFI函数修改）正被转换为元组列表，如下所示：

class FFITuple(Structure):
    _fields_ = [("a", c_uint32),
                ("b", c_uint32)]

def void_array_to_tuple_list(array, _func, _args):
    # this is the errcheck function
    res = cast(array.data, POINTER(FFITuple * array.len))[0]
    res_list = [(i.a, i.b) for i in iter(res)]
    drop_bng_array(array)
    return res_list

这很有效，但__init__和errcheck的转换步骤对于大型列表来说仍然非常慢。首先将列表转换为实现缓冲协议的对象，并使用ctypes FFIArray（或from_buffer，或者甚至{{1}创建from_buffer_copy个对象，是否有任何速度优势？然后对返回的memmove对象执行反向操作？）

Answer 1

将列表转换为array并使用from_buffer导致对象创建时间减少47.5％（测试脚本现在运行204ms而387ms运行，对于百万项目列表）：

FFIArray(Structure):
    _fields_ = [("data", c_void_p),
                ("len", c_size_t)]

    @classmethod
    def from_param(cls, seq):
        return seq if isinstance(seq, cls) else cls(seq)

    def __init__(self, seq, data_type = c_float):
        array_type = data_type * len(seq)
        try:
            raw_seq = array_type.from_buffer(seq.astype(np.float32))
        except (TypeError, AttributeError):
            try:
                raw_eq = array_type.from_buffer_copy(seq.astype(np.float32))
            except (TypeError, AttributeError):
                raw_seq = array_type.from_buffer(array('f', seq))
        self.data = cast(raw_seq, c_void_p)
        self.len = len(seq)

此外，修改FFI函数以返回两个序列，而不是将它们明确地组合成元组，导致了大的加速：

class ResTuple(Structure):
    """ Container for returned FFI data """
    _fields_ = [("e", FFIArray),
                ("n", FFIArray)]


def void_arrays_to_lists(restuple, _func, _args):
    """ Convert the lon, lat --> BNG FFI result to Python data structures """
    eastings = POINTER(c_uint32 * restuple.e.len).from_buffer_copy(restuple.e)[0]
    northings = POINTER(c_uint32 * restuple.n.len).from_buffer_copy(restuple.n)[0]
    res_list = [list(eastings), list(northings)]
    drop_bng_array(restuple.e, restuple.n)
    return res_list

然后我们使用新类稍微修改我们的设置：

convert_bng = lib.convert_to_bng_threaded
convert_bng.argtypes = (FFIArray, FFIArray)
convert_bng.restype = ResTuple
convert_bng.errcheck = void_arrays_to_lists
# cleanup
drop_bng_array = lib.drop_int_array
drop_bng_array.argtypes = (FFIArray, FFIArray)
drop_bng_array.restype = None

测试脚本

import _BNG_FFIArray
import pyproj
import numpy as np

N = 55.811741
E = 1.768960
S = 49.871159
W = -6.379880

bng = pyproj.Proj(init='epsg:27700')
wgs84 = pyproj.Proj(init='epsg:4326')
num_coords = 1000000


def test_speed(lon_ls):
    lon_obj = _BNG_FFIArray(lon_ls)


lons = list(np.random.uniform(W, E, [num_coords]))
res = test_speed(lons)

使用缓冲区更快地转换到ctypes对象和从ctypes对象转换？

1 个答案:

测试脚本