Question

我正在使用https://google.github.io/seq2seq/的代码训练seq2seq NMT（神经机器翻译）。在我中断训练过程后，重新启动的过程变得非常慢（从1.2步/秒到0.07步/秒）。有没有其他人有这种经历？我该如何调试？我已经运行了好几个星期，真的不想放弃......谢谢！〜

正常训练的最后一行，

INFO:tensorflow:loss = 0.585205, step = 830853 (79.477 sec)
INFO:tensorflow:global_step/sec: 1.24179
INFO:tensorflow:loss = 0.267574, step = 830953 (80.529 sec)

超级慢训练的前几行...

INFO:tensorflow:global_step/sec: 0.0746058
INFO:tensorflow:loss = 0.554718, step = 830854 (1340.379 sec)

Answer 1

啊，仔细浏览日志后，似乎有一个关于Cuda的最新更新产生了问题。自动更新现在稍后删除了问题。这是日志..

2017-06-02 22:47:22.684229: E tensorflow/stream_executor/cuda/cuda_driver.cc:405] failed call to cuInit: CUDA_ERROR_NO_DEVICE
2017-06-02 22:47:22.684260: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: POSingularity
2017-06-02 22:47:22.684276: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: POSingularity
2017-06-02 22:47:22.684305: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 375.66.0
2017-06-02 22:47:22.684331: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:369] driver version file contents: """NVRM version: NVIDIA UNIX x86_64 Kernel Module  375.51  Wed Mar 22 10:26:12 PDT 
2017
GCC version:  gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 
"""
2017-06-02 22:47:22.684352: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 375.51.0
2017-06-02 22:47:22.684368: E tensorflow/stream_executor/cuda/cuda_diagnostics.cc:303] kernel version 375.51.0 does not match DSO version 375.66.0 -- cannot find working devices in this configuration

Tensorflow的SEQ2SEQ训练突然超级慢

1 个答案: