具有暗淡[0,0]的相同矩阵已在不同设备之间传输20次

时间:2017-01-23 17:09:15

标签: cntk

在Epoch期间有一定程度,但是一旦Epoch完成,我会看到关于复制空矩阵的警告。通常会导致此警告的原因是什么?

01/23/2017 13:06:49: Epoch[ 1 of 50]-Minibatch[691301-691400]: ce = 0.06757763 * 9404; errs = 1.595% * 9404; time = 14.3775s; samplesPerSecond = 654.1 01/23/2017 13:07:04: Epoch[ 1 of 50]-Minibatch[691401-691500]: ce = 0.08411693 * 9784; errs = 1.962% * 9784; time = 15.1554s; samplesPerSecond = 645.6 01/23/2017 13:07:18: Epoch[ 1 of 50]-Minibatch[691501-691600]: ce = 0.07443892 * 9847; errs = 1.696% * 9847; time = 14.1284s; samplesPerSecond = 697.0 WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. 01/23/2017 13:07:33: Epoch[ 1 of 50]-Minibatch[691601-691700]: ce = 0.07692308 * 9815; errs = 1.854% * 9815; time = 14.4867s; samplesPerSecond = 677.5 WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. 01/23/2017 13:07:48: Epoch[ 1 of 50]-Minibatch[691701-691800]: ce = 0.08028341 * 9809; errs = 1.906% * 9809; time = 14.7772s; samplesPerSecond = 663.8 01/23/2017 13:08:03: Epoch[ 1 of 50]-Minibatch[691801-691900]: ce = 0.09192892 * 10073; errs = 2.214% * 10073; time = 14.8481s; samplesPerSecond = 678.4 01/23/2017 13:08:17: Epoch[ 1 of 50]-Minibatch[691901-692000]: ce = 0.07414725 * 9616; errs = 1.841% * 9616; time = 14.9059s; samplesPerSecond = 645.1 01/23/2017 13:08:32: Finished Epoch[ 1 of 50]: [Training] ce = 0.08177092 * 67573150; errs = 1.962% * 67573150; totalSamplesSeen = 67573150; learningRatePerSample = 0.0020000001; epochTime=104968s WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times. WARNING: The same matrix with dim [0, 0] has been transferred between different devices for 20 times.

1 个答案:

答案 0 :(得分:0)

我从未在空矩阵中看到过这种情况,但是当矩阵具有非零维度时,这是由GPU中不支持的操作引起的(通常是在前向或后向传递中涉及稀疏矩阵的计算)。该消息通常可以由长时间重复触发,其中在每个步骤上麻烦的操作继续向所讨论的矩阵添加一个GPU-CPU-GPU往返。