xarray.open_mfdataset“ ValueError:轴与数组不匹配”

时间:2019-07-15 19:28:08

标签: python dask netcdf python-xarray

我正在使用python xarray的open_mfdataset打开一个分布在多个HDF5文件中的数据集。我有2个数据集,结构上应该几乎相同。第一个数据集工作正常,但是当我尝试打开另一个数据集时,出现错误:

ValueError: axes don't match array

我不知道是什么原因引起的错误,每个Google搜索结果都涉及有关神经网络的问题,这些神经网络似乎会跳出相同的错误,但原因有所不同。

编辑:我应该分享一下,发生错误的行如下所示:

df = xr.open_mfdataset("/some/directory/*.h5", concat_dim='TIME')

我会共享文件,但是它们很大,我不确定这样做的合法性,也不确定在没有文件的情况下如何重现问题。

编辑2:

我认为我找到了问题。数据的(x,y,z)尺寸为(300,300,60)。但是,我使用的模型对x和y使用相同的命名维,因为它的nx和ny数量相同(这是非常愚蠢的imo)。因此,文件标题指出:

-> % ncdump -h icefix-A-2014-02-05-120000-g1.h5 
netcdf icefix-A-2014-02-05-120000-g1 {
dimensions:
        phony_dim_0 = 300 ;
        phony_dim_1 = 60 ;
        phony_dim_2 = 2 ;
        phony_dim_3 = 5 ;
        phony_dim_4 = 11 ;

3D变量看起来像

float CCP(phony_dim_1, phony_dim_0, phony_dim_0) ;

我认为xarray在处理具有重复命名维度的变量时遇到问题。除了改变模型输出文件的方式之外,我不确定是否有办法解决此问题。

完整的回溯如下:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-cb4a12bd492e> in <module>
----> 1 control_d1 = xr.open_mfdataset(datadir + "feb2014_control/icefix*g1.h5", concat_dim='TIME')

~/anaconda3/lib/python3.7/site-packages/xarray/backends/api.py in open_mfdataset(paths, chunks, concat_dim, compat, preprocess, engine, lock, data_vars, coords, autoclose, parallel, **kwargs)
    717             data_vars=data_vars, coords=coords,
    718             infer_order_from_coords=infer_order_from_coords,
--> 719             ids=ids)
    720     except ValueError:
    721         for ds in datasets:

~/anaconda3/lib/python3.7/site-packages/xarray/core/combine.py in _auto_combine(datasets, concat_dims, compat, data_vars, coords, infer_order_from_coords, ids)
    551     # Repeatedly concatenate then merge along each dimension
    552     combined = _combine_nd(combined_ids, concat_dims, compat=compat,
--> 553                            data_vars=data_vars, coords=coords)
    554     return combined
    555 

~/anaconda3/lib/python3.7/site-packages/xarray/core/combine.py in _combine_nd(combined_ids, concat_dims, data_vars, coords, compat)
    473                                                          data_vars=data_vars,
    474                                                          coords=coords,
--> 475                                                          compat=compat)
    476     combined_ds = list(combined_ids.values())[0]
    477     return combined_ds

~/anaconda3/lib/python3.7/site-packages/xarray/core/combine.py in _auto_combine_all_along_first_dim(combined_ids, dim, data_vars, coords, compat)
    491         datasets = combined_ids.values()
    492         new_combined_ids[new_id] = _auto_combine_1d(datasets, dim, compat,
--> 493                                                     data_vars, coords)
    494     return new_combined_ids
    495 

~/anaconda3/lib/python3.7/site-packages/xarray/core/combine.py in _auto_combine_1d(datasets, concat_dim, compat, data_vars, coords)
    509         concatenated = [_auto_concat(list(ds_group), dim=dim,
    510                                      data_vars=data_vars, coords=coords)
--> 511                         for id, ds_group in grouped_by_vars]
    512     else:
    513         concatenated = datasets

~/anaconda3/lib/python3.7/site-packages/xarray/core/combine.py in <listcomp>(.0)
    509         concatenated = [_auto_concat(list(ds_group), dim=dim,
    510                                      data_vars=data_vars, coords=coords)
--> 511                         for id, ds_group in grouped_by_vars]
    512     else:
    513         concatenated = datasets

~/anaconda3/lib/python3.7/site-packages/xarray/core/combine.py in _auto_concat(datasets, dim, data_vars, coords)
    367                                  'explicitly')
    368             dim, = concat_dims
--> 369         return concat(datasets, dim=dim, data_vars=data_vars, coords=coords)
    370 
    371 

~/anaconda3/lib/python3.7/site-packages/xarray/core/combine.py in concat(objs, dim, data_vars, coords, compat, positions, indexers, mode, concat_over)
    118         raise TypeError('can only concatenate xarray Dataset and DataArray '
    119                         'objects, got %s' % type(first_obj))
--> 120     return f(objs, dim, data_vars, coords, compat, positions)
    121 
    122 

~/anaconda3/lib/python3.7/site-packages/xarray/core/combine.py in _dataset_concat(datasets, dim, data_vars, coords, compat, positions)
    303         if k in concat_over:
    304             vars = ensure_common_dims([ds.variables[k] for ds in datasets])
--> 305             combined = concat_vars(vars, dim, positions)
    306             insert_result_variable(k, combined)
    307 

~/anaconda3/lib/python3.7/site-packages/xarray/core/variable.py in concat(variables, dim, positions, shortcut)
   2083         along the given dimension.
   2084     """
-> 2085     variables = list(variables)
   2086     if all(isinstance(v, IndexVariable) for v in variables):
   2087         return IndexVariable.concat(variables, dim, positions, shortcut)

~/anaconda3/lib/python3.7/site-packages/xarray/core/combine.py in ensure_common_dims(vars)
    296                 common_shape = tuple(non_concat_dims.get(d, dim_len)
    297                                      for d in common_dims)
--> 298                 var = var.set_dims(common_dims, common_shape)
    299             yield var
    300 

~/anaconda3/lib/python3.7/site-packages/xarray/core/variable.py in set_dims(self, dims, shape)
   1209         expanded_var = Variable(expanded_dims, expanded_data, self._attrs,
   1210                                 self._encoding, fastpath=True)
-> 1211         return expanded_var.transpose(*dims)
   1212 
   1213     def _stack_once(self, dims, new_dim):

~/anaconda3/lib/python3.7/site-packages/xarray/core/variable.py in transpose(self, *dims)
   1152             return self.copy(deep=False)
   1153 
-> 1154         data = as_indexable(self._data).transpose(axes)
   1155         return type(self)(dims, data, self._attrs, self._encoding,
   1156                           fastpath=True)

~/anaconda3/lib/python3.7/site-packages/xarray/core/indexing.py in transpose(self, order)
   1210 
   1211     def transpose(self, order):
-> 1212         return self.array.transpose(order)
   1213 
   1214 

~/anaconda3/lib/python3.7/site-packages/dask/array/core.py in transpose(self, *axes)
   1331         See Also
   1332         --------
-> 1333         da.store
   1334         h5py.File.create_dataset
   1335         """

~/anaconda3/lib/python3.7/site-packages/dask/array/routines.py in transpose(a, axes)
    136 
    137 
--> 138 @derived_from(np)
    139 def swapaxes(a, axis1, axis2):
    140     if axis1 == axis2:

ValueError: axes don't match array

0 个答案:

没有答案