我正在使用 bert-base-uncased
在 PyTorch 中创建实体提取模型,但是当我尝试运行该模型时出现此错误:
Some weights of the model checkpoint at D:\Transformers\bert-entity-extraction\input\bert-base-uncased_L-12_H-768_A-12 were not used when initializing BertModel:
['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias',
'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight',
'cls.predictions.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
下载了其他文件
以下是我的模型的代码:
import config
import torch
import transformers
import torch.nn as nn
def loss_fn(output, target, mask, num_labels):
lfn = nn.CrossEntropyLoss()
active_loss = mask.view(-1) == 1
active_logits = output.view(-1, num_labels)
active_labels = torch.where(
active_loss,
target.view(-1),
torch.tensor(lfn.ignore_index).type_as(target)
)
loss = lfn(active_logits, active_labels)
return loss
class EntityModel(nn.Module):
def __init__(self, num_tag, num_pos):
super(EntityModel, self).__init__()
self.num_tag = num_tag
self.num_pos = num_pos
self.bert = transformers.BertModel.from_pretrained(config.BASE_MODEL_PATH)
self.bert_drop_1 = nn.Dropout(p = 0.3)
self.bert_drop_2 = nn.Dropout(p = 0.3)
self.out_tag = nn.Linear(768, self.num_tag)
self.out_pos = nn.Linear(768, self.num_pos)
def forward(self, ids, mask, token_type_ids, target_pos, target_tag):
o1, _ = self.bert(ids,
attention_mask = mask,
token_type_ids = token_type_ids)
bo_tag = self.bert_drop_1(o1)
bo_pos = self.bert_drop_2(o1)
tag = self.out_tag(bo_tag)
pos = self.out_pos(bo_pos)
loss_tag = loss_fn(tag, target_tag, mask, self.num_tag)
loss_pos = loss_fn(pos, target_pos, mask, self.num_pos)
loss = (loss_tag + loss_pos) / 2
return tag, pos, loss
print("model.py run success!")
答案 0 :(得分:0)
正如@cronoik 和提到的 here ,这不是错误,但此警告意味着在您的训练期间,您没有使用池化器来计算损失。因此,如果是这种情况,则无需担心。
您可以通过以下方式设置此警告:
from transformers import logging
logging.set_verbosity_warning()
答案 1 :(得分:0)
正如上面那个人所建议的,如果你不想每次都看到这个,我知道我不想,添加以下内容:
`从转换器导入日志
logging.set_verbosity_error()`