Question

我有一个要并行计算的2D（4950，4950）dask数组。使用链接：https://docs.dask.org/en/latest/delayed-best-practices.html#don-t-call-dask-delayed-on-other-dask-collections

print(da.shape)
partitions = da.to_delayed()
print(partitions)
delayed_values = [dask.delayed(funct)(part) for part in partitions]
print(delayed_values)

我得到的结果是：

(4950, 4950)
[[Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 0, 0))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 0, 1))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 0, 2))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 0, 3))]
 [Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 1, 0))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 1, 1))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 1, 2))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 1, 3))]
 [Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 2, 0))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 2, 1))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 2, 2))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 2, 3))]
 [Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 3, 0))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 3, 1))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 3, 2))
  Delayed(('gt-f3b8d1635832fc9b88447def18b4b7d0', 3, 3))]]
[Delayed('funct-c0044e9f-4b8e-4d02-b364-f6a483eaae2f'), 
 Delayed('funct-d2d14dcd-6f0a-4198-b999-221b0609bcaa'), 
 Delayed('funct-1951008c-14f4-43da-bbc1-443e90aae029'), 
 Delayed('funct-a254e3ba-2d45-45f8-bae4-85ba8c37a32f')]

我想找出每个分区的行索引（第一个和最后一个索引），以将每个索引的计算结果保存在最终输出文件中。

我找不到很多与分区有关的文档，非常感谢能够帮助查找行索引的任何帮助/链接。

Answer 1

对于Dask数组，您想查看.chunks属性。特别是我认为您可能想要

之类的东西

[np.cumsum(c) for c in x.chunks]

有关更多信息，请参见https://docs.dask.org/en/latest/array-design.html#chunks

如何查找dask数组分区的行索引

1 个答案: