我正在尝试从一个大的.h5文件中提取一些“行”,以创建一个较小的示例文件。
为了确保我的示例看起来像原始文件,我随机提取行。
#Get length of files and prepare samples
source_file = h5py.File(args.data_path, "r")
dataset = source_file['X']
indices = np.sort(np.random.choice(dataset.shape[0],args.nb_rows))
#checking we're extracting a subsample
if args.nb_rows > dataset.shape[0]:
raise ValueError("Can't extract more rows than dataset contains. Dataset has %s rows" % dataset.shape[0] )
target_file = h5py.File(target, "w")
for k in source_file.keys():
dataset = source_file[k]
dataset = dataset[indices,:,:,:]
dest_dataset = target_file.create_dataset(k, shape=(dataset.shape), dtype=np.float32)
dest_dataset.write_direct(dataset)
target_file.close()
source_file.close()
然而,当nb_rows(如10,000)时,我得到TypeError("Indexing elements must be in increasing order")
。索引已排序,所以我认为我不应该得到这个错误。我误解了什么吗?
答案 0 :(得分:1)
我认为你正在重复。
显然,你会在Earnings
案例中得到重复:
args.nb_rows > dataset.shape[0]
但是当数字较小时你仍然可以获得重复:
In [499]: np.random.choice(10, 20)
Out[499]: array([2, 4, 1, 5, 2, 8, 4, 3, 7, 0, 2, 6, 6, 8, 9, 3, 8, 4, 2, 5])
In [500]: np.sort(np.random.choice(10, 20))
Out[500]: array([1, 1, 1, 2, 2, 2, 4, 4, 4, 5, 5, 5, 5, 6, 6, 7, 8, 8, 8, 9])
关闭In [502]: np.sort(np.random.choice(10, 9))
Out[502]: array([0, 0, 1, 1, 1, 5, 5, 9, 9])
:
replace