Hack`kur -v data mnist.yml`用于更简单地概述提供者的数据？

时间：2017-03-25 08:40:32

标签： python deep-learning

问题

当我运行kur -v data mnist.yml时，kur打印出一批样本，这些样本很长，您可以看到或花很长时间滚动到日志消息的开头。

然后我将日志输出存储在kur -v data mnist.yml 2>&1 | tee my.txt的文件中，现在我可以粘贴下面的日志消息

[1;37m[INFO 2017-03-25 16:18:09,717 kur.kurfile:709][0m Parsing source: mnist.yml, included by top-level.[0m
[1;37m[INFO 2017-03-25 16:18:09,724 kur.kurfile:709][0m Parsing source: mnist-defaults.yml, included by mnist.yml.[0m
[1;37m[INFO 2017-03-25 16:18:09,733 kur.kurfile:84][0m Parsing Kurfile...[0m
[1;37m[INFO 2017-03-25 16:18:09,749 kur.__main__:144][0m Preparing data sources for: train[0m
Entry 1/32:
  images: [[[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
...
...

这个输出的问题是它不是我想要的。我通常需要数据概述中的两件事： 1.提供数据的形状 2.打印出的样本

当前kur -v data mnist.yml只打印出一批32个样本。公平地说，kur -vv data mnist.yml确实告诉我们批次中的样本数量。就是这样。

挑战

如何让kur -v data mnist.yml打印出来：

  1. 提供数据的形状
  2. 打印出的样本

    3. 如下所示：

    [INFO 2017-03-25 14:31:14,318 kur.kurfile:760] Parsing source: mnist.yml, included by top-level.
[INFO 2017-03-25 14:31:14,324 kur.kurfile:760] Parsing source: mnist-defaults.yml, included by mnist.yml.
[INFO 2017-03-25 14:31:14,334 kur.kurfile:113] Parsing Kurfile...
[INFO 2017-03-25 14:31:15,465 kur.__main__:290] A single batch consists of: {'labels': (32, 10), 'images': (32, 28, 28, 1)}
[INFO 2017-03-25 14:31:15,465 kur.__main__:294] There are total 32 samples in a single batch.
[INFO 2017-03-25 14:31:15,466 kur.__main__:295] Let's have a look a single sample
Entry 1/32:
  images: [[[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.37130019]
  [ 0.37130019]
  [ 0.86933935]
  [ 0.61835897]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.12031978]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.12031978]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.61835897]
  [ 0.37130019]
  [ 0.12031978]
  [ 0.61835897]
  [ 0.86933935]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.37130019]
  [ 0.86933935]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.12031978]
  [ 0.61835897]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.61835897]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.61835897]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.61835897]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.61835897]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.37130019]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.12031978]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.12031978]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.37130019]
  [ 0.37130019]
  [ 0.37130019]
  [ 0.37130019]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.37130019]
  [ 0.86933935]
  [ 0.61835897]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.12031978]
  [ 0.61835897]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.61835897]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.61835897]
  [ 0.86933935]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.61835897]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.37130019]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.37130019]
  [ 0.61835897]
  [ 0.86933935]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.37130019]
  [ 0.86933935]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.86933935]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.37130019]
  [ 0.86933935]
  [ 0.86933935]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.12031978]
  [ 0.86933935]
  [ 0.61835897]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.61835897]
  [ 0.86933935]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.61835897]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.12031978]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.37130019]
  [ 0.61835897]
  [ 0.86933935]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.61835897]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.37130019]
  [ 0.37130019]
  [ 0.37130019]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.37130019]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [ 0.37130019]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.86933935]
  [ 0.61835897]
  [ 0.37130019]
  [ 0.37130019]
  [ 0.12031978]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]

 [[-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]
  [-0.13066062]]]
  labels: [ 0.  0.  0.  0.  0.  0.  1.  0.  0.  0.]

2 个答案:

答案 0 :(得分：0)

一种方法是破解__main__.py

替换the original code：

num_entries = len(batch[keys[0]])
for entry in range(num_entries):
    print('Entry {}/{}:'.format(entry+1, num_entries))
    for k in keys:
        print('  {}: {}'.format(k, batch[k][entry]))

使用this new code：

#### What is the shape of each key's values
batch_shape = {}
for k in keys:
    batch_shape[k] = batch[k].shape
logger.info("A single batch consists of: %s", batch_shape)

### count inside the first element of batch dictionary, assign to num_entries
num_entries = len(batch[keys[0]])
logger.info("There are total %s samples in a single batch.", num_entries)
logger.info("Let's have a look a single sample")

### print out the whole batch, but one sample pair at a time, each sample pair like (key1:value, key2: value)
# for entry in range(0, num_entries)
### Here only print out the first and last sample of a batch
# for entry in range(0, num_entries, num_entries-1):
for entry in range(1):
    ## print Entry entry_idx/total_entries
    print('Entry {}/{}:'.format(entry+1, num_entries))
    ## for every batch_key
    for k in keys:
        # print: batch_key: this batch's first item
        print('  {}: {}'.format(k, batch[k][entry]))

答案 1 :(得分：0)

这是另一种输出更简单数据视图的方法：

In [17]: !kur -v data --single mnist.yml
[INFO 2017-03-29 11:34:42,369 kur.kurfile:712] Parsing source: mnist.yml, included by top-level.
[INFO 2017-03-29 11:34:42,376 kur.kurfile:712] Parsing source: mnist-defaults.yml, included by mnist.yml.
[INFO 2017-03-29 11:34:42,386 kur.kurfile:85] Parsing Kurfile...
[INFO 2017-03-29 11:34:42,407 kur.__main__:147] Preparing data sources for: train
[INFO 2017-03-29 11:34:43,309 kur.__main__:186] We are training with 15 batches using 480 samples of data
[INFO 2017-03-29 11:34:43,309 kur.__main__:190] A single batch consists of: {'labels': (32, 10), 'images': (32, 28, 28, 1)}
Entry 1/32:
  images: [[[-0.13066062]
  [-0.13066062]
相关问题
最新问题