在多个GPU上运行NiftyNet Promise12演示出错

时间:2018-04-05 14:50:16

标签: tensorflow niftynet

我正在尝试复制Promise12 demo in NiftyNet。使用TensorFlow 1.3 GPU版本安装NiftyNet 0.2.2。

下载PROMISE12 data并解压后,我修改了NiftyNet的promise12_demo_train_config.ini以反映我的数据路径(这是唯一的更改)。在1 GPU上运行是成功的。

接下来,我将the [SYSTEM] section of the config更改为:

[SYSTEM]
cuda_devices = ""
num_threads = 2
num_gpus = 3
model_dir = /promise12_model

想法是利用3个GPU。但是,这会产生以下错误,重复多次,看起来像每个输入图像一次:

InvalidArgumentError (see above for traceback): Duplicate tag imgseg/image/0 found in summary inputs
     [[Node: MergedOutputs/Merge/MergeSummary = MergeSummary[N=4, _device="/job:localhost/replica:0/task:0/cpu:0"](worker_0/DenseVNet/PyFunc, worker_1/DenseVNet/PyFunc, worker_2/DenseVNet/PyFunc, MergedOutputs/loss_device_average_)]]
     [[Node: worker_1/gradients/worker_1/loss_function_1/Mean_grad/Shape_3/_3202 = _HostSend[T=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:1", send_device="/job:localhost/replica:0/task:0/gpu:1", send_device_incarnation=1, tensor_name="edge_53477_worker_1/gradients/worker_1/loss_function_1/Mean_grad/Shape_3", _device="/job:localhost/replica:0/task:0/gpu:1"](worker_1/gradients/worker_1/loss_function_1/Mean_grad/Shape_3)]]

我怀疑这是预期的结果,但是当使用多个GPU时,还有其他需要更改的设置?

1 个答案:

答案 0 :(得分:1)

此处讨论了问题/解决方案:https://github.com/NifTK/NiftyNet/issues/50