Question

对于单词列表，我想获取它们的快速文本向量并将其保存为相同的“ word2vec” .txt格式（txt格式的word + space + vector）。

这就是我所做的：

dict = open("word_list.txt","r") #the list of words I have

path = "cc.en.300.bin" 

model = load_facebook_model(path)

vectors = []

words =[] 

for word in dict: 
    vectors.append(model[word])
    words.append(word)

vectors_array = np.array(vectors)

*我想获取列表“ words”和nd.array“ vectors_array”并保存为原始.txt格式。

我尝试使用gensim“ _save_word2vec_format”中的函数：

def _save_word2vec_format(fname, vocab, vectors, fvocab=None, binary=False, total_vec=None):
    """Store the input-hidden weight matrix in the same format used by the original
    C word2vec-tool, for compatibility.
    Parameters
    ----------
    fname : str
        The file path used to save the vectors in.
    vocab : dict
        The vocabulary of words.
    vectors : numpy.array
        The vectors to be stored.
    fvocab : str, optional
        File path used to save the vocabulary.
    binary : bool, optional
        If True, the data wil be saved in binary word2vec format, else it will be saved in plain text.
    total_vec : int, optional
        Explicitly specify total number of vectors
        (in case word vectors are appended with document vectors afterwards).
    """
    if not (vocab or vectors):
        raise RuntimeError("no input")
    if total_vec is None:
        total_vec = len(vocab)
    vector_size = vectors.shape[1]
    if fvocab is not None:
        logger.info("storing vocabulary in %s", fvocab)
        with utils.open(fvocab, 'wb') as vout:
            for word, vocab_ in sorted(iteritems(vocab), key=lambda item: -item[1].count):
                vout.write(utils.to_utf8("%s %s\n" % (word, vocab_.count)))
    logger.info("storing %sx%s projection weights into %s", total_vec, vector_size, fname)
    assert (len(vocab), vector_size) == vectors.shape
    with utils.open(fname, 'wb') as fout:
        fout.write(utils.to_utf8("%s %s\n" % (total_vec, vector_size)))
        # store in sorted order: most frequent words at the top
        for word, vocab_ in sorted(iteritems(vocab), key=lambda item: -item[1].count):
            row = vectors[vocab_.index]
            if binary:
                row = row.astype(REAL)
                fout.write(utils.to_utf8(word) + b" " + row.tostring())
            else:
                fout.write(utils.to_utf8("%s %s\n" % (word, ' '.join(repr(val) for val in row))))

但是我得到了错误：

INFO:gensim.models._fasttext_bin:loading 2000000 words for fastText model from cc.en.300.bin
INFO:gensim.models.word2vec:resetting layer weights
INFO:gensim.models.word2vec:Updating model with new vocabulary
INFO:gensim.models.word2vec:New added 2000000 unique words (50% of original 4000000) and increased the count of 2000000 pre-existing words (50% of original 4000000)
INFO:gensim.models.word2vec:deleting the raw counts dictionary of 2000000 items
INFO:gensim.models.word2vec:sample=1e-05 downsamples 6996 most-common words
INFO:gensim.models.word2vec:downsampling leaves estimated 390315457935 word corpus (70.7% of prior 552001338161)
INFO:gensim.models.fasttext:loaded (4000000, 300) weight matrix for fastText model from cc.en.300.bin
trials.py:42: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).
  vectors.append(model[word])
INFO:__main__:storing 8664x300 projection weights into arrays_to_txt_oct3.txt
loading the model for: en
finish loading the model for: en
len(vectors): 8664
len(words):  8664
shape of vectors_array (8664, 300)
mission launched!
Traceback (most recent call last):
  File "trials.py", line 102, in <module>
    _save_word2vec_format(YOUR_VEC_FILE_PATH, words, vectors_array, fvocab=None, binary=False, total_vec=None)
  File "trials.py", line 89, in _save_word2vec_format
    for word, vocab_ in sorted(iteritems(vocab), key=lambda item: -item[1].count):
  File "/cs/snapless/oabend/tailin/transdiv/lib/python3.7/site-packages/six.py", line 589, in iteritems
    return iter(d.items(**kw))
AttributeError: 'list' object has no attribute 'items'

我知道它与函数中的第二个参数有关，但是我不知道如何将单词列表放入字典对象中？

我尝试过这样做：

#convert list of words into a dictionary
words_dict = {i:x for i,x in enumerate(words)}

但仍然收到错误消息：

Traceback (most recent call last):
  File "trials.py", line 99, in <module>
    _save_word2vec_format(YOUR_VEC_FILE_PATH, dict, vectors_array, fvocab=None, binary=False, total_vec=None)
  File "trials.py", line 77, in _save_word2vec_format
    total_vec = len(vocab)
TypeError: object of type '_io.TextIOWrapper' has no len()

我不明白如何以正确的格式插入单词列表...

Answer 1

您可以直接导入并重新使用Gensim KeyedVectors类，以将自己的（子）词向量集组合为KeyedVectors的一个实例，然后使用其.save_word2vec_format()方法

例如，大致这应该可行：

from gensim.models import KeyedVectors

words_file = open("word_list.txt","r")  # your word-list as a text file
words_list = list(words_file)  # reads each line of file into a new `list` object

fasttext_path = "cc.en.300.bin" 
model = load_facebook_model(path)

kv = KeyedVectors(vector_size=model.wv.vector_size)  # new empty KV object

vectors = [] 
for word in words_list: 
    vectors.append(model[word])  # vectors for words_list, in same order

kv.add(words_list, vectors)  # adds those keys (words) & vectors as batch

kv.save_word2vec_format('my_kv.vec', binary=False)

使用_save_word2vec_format（）以“ word2vec”格式保存预训练的快速文本向量时出现问题

1 个答案: