Question

我下载了经过预训练的文件格式google，并使用官方脚本convert_bert_original_tf_checkpoint_to_pytorch.py将其转换为名称为'pytorch_model.bin'的火炬版：

current_dir = os.path.dirname(__file__) # (1)
model_folder = os.path.join(current_dir, 'chinese_L-12_H-768_A-12')
checkpoint = os.path.join(model_folder, 'bert_model.ckpt')
configer = os.path.join(model_folder, 'bert_config.json')
save_to = os.path.join(current_dir, 'torch bert model', 'pytorch_model.bin')
convert_tf_checkpoint_to_pytorch(checkpoint, configer, save_to)

然后，我将“ bert_config.json”重命名为“ config.json”，并将“ pytorch_model.bin”，“ config.json”和“ vocab.txt”放入相同的文件夹“ torch_bert_model”中，以便加载该手电筒'from_pertained（）'方法的模型：

current_dir = os.path.dirname(__file__)
model_folder = os.path.join(current_dir, 'torch_bert_model')
# configer_file = os.path.join(model_folder, 'config.json')
vocab_file = os.path.join(model_folder, 'vocab.txt')
model_file = os.path.join(model_folder, 'pytorch_model.bin')

tokenizer = BertTokenizer.from_pretrained(vocab_file)
# bert_config = BertConfig.from_json_file(configer_file)
model = TFBertModel.from_pretrained(model_file) # Automatically loads the config

运行此代码时，发生了“ UnicodeDecodeError”：

Traceback (most recent call last):
File "c:/Users/hap/Desktop/Projects/cgt/BERT.py", line 47, in <module>
  model = TFBertModel.from_pretrained(model_file)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site- packages\transformers\modeling_tf_utils.py", line 529, in from_pretrained
  config, model_kwargs = cls.config_class.from_pretrained(
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py", line 311, in from_pretrained
  config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py", line 354, in get_config_dict
  config_dict = cls._dict_from_json_file(resolved_config_file)
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_utils.py", line 436, in _dict_from_json_file
  text = reader.read()
File "C:\Users\hap\AppData\Local\Programs\Python\Python38\lib\codecs.py", line 322, in decode
  (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte

我在哪里错了？任何建议，不胜感激。谢谢！

使用转换器程序包加载BertModel时遇到“ UnicodeDecodeError”

0 个答案: