Question

我使用Pytorch和BERT训练模型。 Everithing可以在一个GPU上很好地工作，但是当我尝试使用多个GPU时，我得到一个错误：

undefined

有人可以帮助我我所缺少的内容以及应该如何解决？

这是我的培训代码：

ValueError                                Traceback (most recent call last)
<ipython-input-168-507223f9879c> in <module>()
     92         # single value; the `.item()` function just returns the Python value
     93         # from the tensor.
---> 94         total_loss += loss.item()
     95 
     96         # Perform a backward pass to calculate the gradients.

ValueError: only one element tensors can be converted to Python scalars

这是我的模型代码：

import random
seed_val = 42
random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)
loss_values = []
for epoch_i in range(0, epochs):

    t0 = time.time()
    total_loss = 0
    for step, batch in enumerate(train_dataloader):
        if step % 40 == 0 and not step == 0:
            elapsed = format_time(time.time() - t0)

        b_input_ids = batch[0].to(device).long() 
        b_input_mask = batch[1].to(device).long()
        b_labels = batch[2].to(device).long()
        model.zero_grad()        
        outputs = model(b_input_ids, 
                    token_type_ids=None, 
                    attention_mask=b_input_mask, 
                    labels=b_labels)

        loss = outputs[0]
        total_loss += loss.item()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        scheduler.step()
    avg_train_loss = total_loss / len(train_dataloader)            
    loss_values.append(avg_train_loss)
    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Training epcoh took: {:}".format(format_time(time.time() - t0)))

Answer 1

loss = outputs[0]丢失后，loss是一个多元素张量，大小是GPU的数量。

您可以改用loss = outputs[0].mean()。

多GPU训练中的损失功能（PyTorch）

1 个答案: