h5py集体例程的参数不一致

时间:2018-12-04 10:51:59

标签: python mpi h5py parallel-io

我有4个mpi进程,每个进程将写入多个连续的numpy数组。 data_per_rank 是一个二维numpy数组,每个mpi进程都应将其拆分为 inner_splits numpy子数组,并将每个数组写入不同的hdf5组。

inner_splits = 3
data_splits = np.array_split(data_per_rank, inner_splits, axis=0)

for i in range(size):
    for j in range(inner_splits):
        dset.append(f.create_dataset('p{0}_{1}'.format(i, j), data_splits[j].shape, dtype='float64'))

for i in range(inner_splits):
   dset[(rank*inner_splits)+i][:,:] = data_splits[i]

for i in range(size):
    del dset[i]

f.close()

这会产生以下错误:

File "/gpfs/projects/bsc19/bsc19004/matmul_separated.py", line 62, in write f.close()
File "/apps/PYTHON/2.7.13/INTEL/lib/python2.7/site-packages/h5py-2.7.0-py2.7-linux-x86_64.egg/h5py/_hl/files.py", line 299, in close
   h5i.dec_ref(id_)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/gpfs/apps/MN4/PYTHON/SRC/modules/2.7.13/h5py-.7.0_2/h5py/_objects.c:318)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/gpfs/apps/MN4/PYTHON/SRC/modules/2.7.13/h5py-.7.0_2/h5py/_objects.c:314)
File "h5py/h5i.pyx", line 150, in h5py.h5i.dec_ref(gpfs/apps/MN4/PYTHON/SRC/modules/2.7.13/h5py-2.7.0_2/h5py/h5i.c:2836)
RuntimeError: Can't decrement id ref count (Invalid argument, error stack:
mpi_file_set_size(82): inconsistent arguments to collective routine )

某些子数组的形状为(2731,32768),而其他子数组的形状为(2730,32768)。我确保在编写过程中将正确的data_split写入相应的dset处理程序。

有人可以向我解释这里出了什么问题吗?此错误消息是什么意思?哪些论据不一致?

预先感谢

0 个答案:

没有答案