Question

我想在不使用字标记化的情况下使用spacy的POS标记，NER和依赖项解析。实际上，我的输入是代表句子的标记列表，我想尊重用户的标记化。这可能是spacy或任何其他NLP包吗？

目前，我正在使用这个基于spacy的函数以Conll格式输出一个句子（一个unicode字符串）：

import spacy
nlp = spacy.load('en')
def toConll(string_doc, nlp):
   doc = nlp(string_doc)
   block = []
   for i, word in enumerate(doc):
          if word.head == word:
                  head_idx = 0
          else:
                  head_idx = word.head.i - doc[0].i + 1
          head_idx = str(head_idx)
          line = [str(i+1), str(word), word.lemma_, word.tag_,
                      word.ent_type_, head_idx, word.dep_]
          block.append(line)
   return block
conll_format = toConll(u"Donald Trump is the new president of the United States of America")

Output:
[['1', 'Donald', u'donald', u'NNP', u'PERSON', '2', u'compound'],
 ['2', 'Trump', u'trump', u'NNP', u'PERSON', '3', u'nsubj'],
 ['3', 'is', u'be', u'VBZ', u'', '0', u'ROOT'],
 ['4', 'the', u'the', u'DT', u'', '6', u'det'],
 ['5', 'new', u'new', u'JJ', u'', '6', u'amod'],
 ['6', 'president', u'president', u'NN', u'', '3', u'attr'],
 ['7', 'of', u'of', u'IN', u'', '6', u'prep'],
 ['8', 'the', u'the', u'DT', u'GPE', '10', u'det'],
 ['9', 'United', u'united', u'NNP', u'GPE', '10', u'compound'],
 ['10', 'States', u'states', u'NNP', u'GPE', '7', u'pobj'],
 ['11', 'of', u'of', u'IN', u'GPE', '10', u'prep'],
 ['12', 'America', u'america', u'NNP', u'GPE', '11', u'pobj']]

我想在输入令牌列表的同时做同样的事情......

Answer 1

您可以针对已经标记化的文本运行Spacy的处理管道。但是，您需要了解基础统计模型是否已经使用某种策略进行了标记化的参考语料库进行了训练，如果您的标记化策略明显不同，您可能会期望性能下降。

以下是使用Spacy 2.0.5和Python 3的方法。如果使用Python 2，您可能需要使用unicode文字。

import spacy; nlp = spacy.load('en_core_web_sm')
# spaces is a list of boolean values indicating if subsequent tokens
# are followed by any whitespace
# so, create a Spacy document with your tokenisation
doc = spacy.tokens.doc.Doc(
    nlp.vocab, words=['nuts', 'itch'], spaces=[True, False])
# run the standard pipeline against it
for name, proc in nlp.pipeline:
    doc = proc(doc)

Answer 2

是的，您只需要使用 nlp.tokenizer=nlp.tokenizer.tokens_from_list

import spacy
nlp = spacy.load("en_core_web_sm")
nlp.tokenizer=nlp.tokenizer.tokens_from_list
text="she went to school"
words=text.split()
doc = nlp(words)
for token in doc:   
  token_i = token.i+1
  if token.i==token.head.i: head_i=0
  else: head_i = token.head.i+1
  items=[token_i,token.text, token.lemma_, token.tag_, token.pos_, "_", head_i, token.dep_,"_","_"]
  print(items)

输出：

[1, 'she', '-PRON-', 'PRP', 'PRON', '_', 2, 'nsubj', '_', '_']
[2, 'went', 'go', 'VBD', 'VERB', '_', 0, 'ROOT', '_', '_']
[3, 'to', 'to', 'IN', 'ADP', '_', 2, 'prep', '_', '_']
[4, 'school', 'school', 'NN', 'NOUN', '_', 3, 'pobj', '_', '_']

spacy是否将令牌列表作为输入？

2 个答案: