完整的堆栈跟踪为:
File "model_2_classes.py", line 339, in <module>
iou_period=10)
File "/home/ec2-user/SageMaker/Mask-RCNN-Greenhouses/model.py", line 2527, in train
use_multiprocessing=True,
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/keras/engine/training.py", line 1415, in fit_generator
initial_epoch=initial_epoch)
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/keras/engine/training_generator.py", line 213, in fit_generator
class_weight=class_weight)
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/keras/engine/training.py", line 1215, in train_on_batch
outputs = self.train_function(ins)
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2666, in __call__
return self._call(inputs)
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2636, in _call
fetched = self._callable_fn(*array_vals)
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1382, in __call__
run_metadata_ptr)
File "/home/ec2-user/anaconda3/envs/JupyterSystemEnv/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Reduction axis 1 is empty in shape [0,0]
[[Node: proposal_targets/ArgMax_3 = ArgMax[T=DT_FLOAT, Tidx=DT_INT32, output_type=DT_INT64, _device="/job:localhost/replica:0/task:0/device:GPU:0"](proposal_targets/GatherV2_40, training/SGD/gradients/rpn_class_loss/Mean_grad/Maximum)]]
[[Node: training/SGD/gradients/mrcnn_class_conv1/convolution_grad/Conv2DBackpropInput-0-0-TransposeNCHWToNHWC-LayoutOptimizer/_6399 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_11947...tOptimizer", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
在训练过程中,我在随机点出现此错误。所以我认为我的图像不是问题。另外,我尝试做use_multiprocessing=False
。
使用的模型是Mask-RCNN。 我很乐意提供所需的更多详细信息。