CNTK分布式崩溃 - Beta 7

时间:2017-01-04 15:46:01

标签: python cntk

我正在运行CIFAR 10的变体,以便利用我的数据。

我收到以下错误:

Traceback (most recent call last):
  File "CNTK_Train.py", line 158, in <module>
    checkpoint_path = "C:/projects/RoboLabs/CognitiveServices/ML_Models/DocSuite/Doc_Classify/checkpoints/CNTK_VGG9")
  File "CNTK_Train.py", line 80, in train_and_evaluate
    trainer.save_checkpoint(os.path.join(checkpoint_path + "_{}.dnn".format(current_epoch)))
  File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py34\lib\site-packages\cntk\trainer.py", line 138, in save_checkpoint
    super(Trainer, self).save_checkpoint(filename, _py_dict_to_cntk_dict(external_state))
  File "C:\local\Anaconda3-4.1.1-Windows-x86_64\envs\cntk-py34\lib\site-packages\cntk\cntk_py.py", line 1774, in save_checkpoint
    return _cntk_py.Trainer_save_checkpoint(self, *args)
RuntimeError: Runtime exception

我用于带检查点的训练循环的代码在这里:

while updated:
    data=train_reader.next_minibatch(minibatch_size, input_map=input_map) # fetch minibatch.
    updated=trainer.train_minibatch(data)                                 # update model with it
    progress_printer.update_with_trainer(trainer, with_metric=True)       # log progress
    epoch_index = int(trainer.total_number_of_samples_seen/epoch_size)
    if current_epoch != epoch_index:                                      # new epoch reached
        progress_printer.epoch_summary(with_metric=True)
        current_epoch=epoch_index            
    if current_epoch % 25 == 0:
        trainer.save_checkpoint(os.path.join(checkpoint_path + "_{}.dnn".format(current_epoch)))

欢迎见解。我正在积极调试。

2 个答案:

答案 0 :(得分:0)

是否有可能以Linux风格指定的Windows环境路径格式运行。在Windows上,路径应该类似于'X:\ Repos \ CNTK \ Examples \ Image \ Classification \ ResNet \ Python \ Models \ resnet20_0.dnn'。建议你在传递给save_model的路径字符串中尝试os.path.join而不是hardcode /或\。

答案 1 :(得分:0)

这似乎在最新版本中得到解决,该版本以不同的方式执行检查指向。解决方案是升级您的CNTK版本。使用从版本9开始的会话apis。