在Epoch期间有一定程度,但是一旦Epoch完成,我会看到关于复制空矩阵的警告。通常会导致此警告的原因是什么?
01/23/2017 13:06:49: Epoch[ 1 of 50]-Minibatch[691301-691400]: ce = 0.06757763 * 9404; errs = 1.595% * 9404; time = 14.3775s; samplesPerSecond = 654.1
01/23/2017 13:07:04: Epoch[ 1 of 50]-Minibatch[691401-691500]: ce = 0.08411693 * 9784; errs = 1.962% * 9784; time = 15.1554s; samplesPerSecond = 645.6
01/23/2017 13:07:18: Epoch[ 1 of 50]-Minibatch[691501-691600]: ce = 0.07443892 * 9847; errs = 1.696% * 9847; time = 14.1284s; samplesPerSecond = 697.0
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
01/23/2017 13:07:33: Epoch[ 1 of 50]-Minibatch[691601-691700]: ce = 0.07692308 * 9815; errs = 1.854% * 9815; time = 14.4867s; samplesPerSecond = 677.5
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
01/23/2017 13:07:48: Epoch[ 1 of 50]-Minibatch[691701-691800]: ce = 0.08028341 * 9809; errs = 1.906% * 9809; time = 14.7772s; samplesPerSecond = 663.8
01/23/2017 13:08:03: Epoch[ 1 of 50]-Minibatch[691801-691900]: ce = 0.09192892 * 10073; errs = 2.214% * 10073; time = 14.8481s; samplesPerSecond = 678.4
01/23/2017 13:08:17: Epoch[ 1 of 50]-Minibatch[691901-692000]: ce = 0.07414725 * 9616; errs = 1.841% * 9616; time = 14.9059s; samplesPerSecond = 645.1
01/23/2017 13:08:32: Finished Epoch[ 1 of 50]: [Training] ce = 0.08177092 * 67573150; errs = 1.962% * 67573150; totalSamplesSeen = 67573150; learningRatePerSample = 0.0020000001; epochTime=104968s
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.
答案 0 :(得分:0)
我从未在空矩阵中看到过这种情况,但是当矩阵具有非零维度时,这是由GPU中不支持的操作引起的(通常是在前向或后向传递中涉及稀疏矩阵的计算)。该消息通常可以由长时间重复触发,其中在每个步骤上麻烦的操作继续向所讨论的矩阵添加一个GPU-CPU-GPU往返。