Question

我想对Dask做以下事情：

从HDF5文件加载矩阵
并行化每个条目的计算

这是我的代码：

def blocked_func(x):
    return np.random.random()

with h5py.File(file_path) as f:
    d = f['/data']
    arr = da.from_array(d, chunks=(chunks_row, chunks_col))

    arr2 = arr.map_blocks(blocked_func, dtype='float32').compute()

但是代码会引发以下错误：

File ".../remote_fr_thinkpad/test_big_data.py", line 43, in <module>
    arr2 = arr.map_blocks(blocked_func, dtype='float32').compute()
  File ".../anaconda3/lib/python3.7/site-packages/dask/base.py", line 156, in compute
    (result,) = compute(self, traverse=False, **kwargs)
  File ".../anaconda3/lib/python3.7/site-packages/dask/base.py", line 399, in compute
    return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
  File ".../anaconda3/lib/python3.7/site-packages/dask/base.py", line 399, in <listcomp>
    return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
  File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 779, in finalize
    return concatenate3(results)
  File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 3497, in concatenate3
    chunks = chunks_from_arrays(arrays)
  File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 3327, in chunks_from_arrays
    result.append(tuple([shape(deepfirst(a))[dim] for a in arrays]))
  File ".../anaconda3/lib/python3.7/site-packages/dask/array/core.py", line 3327, in <listcomp>
    result.append(tuple([shape(deepfirst(a))[dim] for a in arrays]))
IndexError: tuple index out of range

我在Google周围搜索，还尝试了dask的gu_func，但这引发了相同的错误。

感谢您的帮助。

Answer 1

map_block期望blocked_func返回与其输入(chunks_row, chunks_col)相同形状的数组，而实际上只返回浮点数。

尝试使用

1）保留形状的函数，例如：

def blocked_func(x):
    return x*2

或

2）告诉map_blocks输出的形状将有所不同：

arr2 = arr.map_blocks(blocked_func, chunks=(1,1), dtype='float32').compute()

，但将输入数组的维数保留在blocked_func中，例如：

def blocked_func(x):
    return np.random.random()[None,None]
    # or like this
    # return np.array([1,1])

Dask map_blocks-IndexError：元组索引超出范围

1 个答案: