Question

对不起，我是GPU训练模型的新手。我正在尝试使用4个GPU和1个cpu（400,000个训练样本）来训练深度学习模型。我正在使用以下代码：

tensorflow.compat.v1.disable_eager_execution()
with tensorflow.device("/cpu:0"):
    initializer = tensorflow.keras.initializers.GlorotNormal()
    inputs1 = Input(shape=(1000, ),name="inputs1")
    input2 = Input(shape=(2, ),name="input2")
    first_dense = Dense(128,kernel_initializer=initializer, )(inputs1)
    second_dense=Dense(8,)(first_dense)
    merge = concatenate([second_dense, input2])
    drop = Dropout(0.1)(merge)
    output = Dense(1, )(drop)
    model = Model(inputs=[inputs1, input2], outputs=output)
parallel_model = multi_gpu_model(model, gpus=4)
parallel_model.compile(loss='mean_squared_error',     optimizer=tensorflow.keras.optimizers.Adam(learning_rate=LR,beta_1=0.9,beta_2=0.999,epsilon=1e-07,amsgrad=False,name="Adam"), metrics=['mse'])
parallel_model.fit([x1_train,x2_train],y_train,validation_data=([x1_test,x2_test],y_test), callbacks=[lr_sched], verbose=2,batch_size=32, epochs=60)
tensorflow.keras.backend.clear_session()

如果有帮助，我也会收到此警告：

WARNING:tensorflow:Model inputs must come from `tf.keras.Input` (thus holding past layer metadata), they cannot be the output of a previous non-Input layer. Here, a tensor specified as input to "model_1" was not an Input tensor, it was generated by layer input2.
Note that input tensors are instantiated via `tensor = tf.keras.Input(shape)`.
The tensor that caused the issue was: input2:0

但是，我只使用每个GPU的7％（我可以使用watch -n0.5 nvidia-smi进行检查），并且训练非常缓慢。如何有效使用所有GPU和CPU并更快地训练模型？

具有多个GPU的深度学习模型的训练非常慢

0 个答案: