Question

我发现这段代码使文字变得有趣该文本被分成句子然后被标记化最后，令牌被词状化了。

我的问题是我不需要执行splitting和tokenize的步骤，因为我已经在我的程序中执行了此操作。
我只是想将词形还原的步骤整合到我的程序中，因为我已经有了一个我必须引用的单词列表。

这是我要集成的程序，没有在词形还原之前发生的步骤。

import java.util.LinkedList;
import edu.stanford.nlp.ling.CoreAnnotations.LemmaAnnotation;
import edu.stanford.nlp.ling.CoreAnnotations.SentencesAnnotation;
import edu.stanford.nlp.ling.CoreLabel;
import edu.stanford.nlp.pipeline.Annotation;
import edu.stanford.nlp.pipeline.StanfordCoreNLP;
import edu.stanford.nlp.util.CoreMap;

public class StanfordLemmatizer {

protected StanfordCoreNLP pipeline;

public StanfordLemmatizer() {
    // Create StanfordCoreNLP object properties, with POS tagging
    // (required for lemmatization), and lemmatization
    Properties props;
    props = new Properties();
    props.put("annotators", "tokenize, ssplit, pos, lemma");

    /*
     * This is a pipeline that takes in a string and returns various analyzed linguistic forms. 
     * The String is tokenized via a tokenizer (such as PTBTokenizerAnnotator), 
     * and then other sequence model style annotation can be used to add things like lemmas, 
     * POS tags, and named entities. These are returned as a list of CoreLabels. 
     * Other analysis components build and store parse trees, dependency graphs, etc. 
     * 
     * This class is designed to apply multiple Annotators to an Annotation. 
     * The idea is that you first build up the pipeline by adding Annotators, 
     * and then you take the objects you wish to annotate and pass them in and 
     * get in return a fully annotated object.
     * 
     *  StanfordCoreNLP loads a lot of models, so you probably
     *  only want to do this once per execution
     */
    this.pipeline = new StanfordCoreNLP(props);
}

public List<String> lemmatize(String documentText)
{
    List<String> lemmas = new LinkedList<String>();
    // Create an empty Annotation just with the given textd
    Annotation document = new Annotation(documentText);
    // run all Annotators on this text
    this.pipeline.annotate(document);
    // Iterate over all of the sentences found
    List<CoreMap> sentences = document.get(SentencesAnnotation.class);
    for(CoreMap sentence: sentences) {
        // Iterate over all tokens in a sentence
        for (CoreLabel token: sentence.get(TokensAnnotation.class)) {
            // Retrieve and add the lemma for each word into the
            // list of lemmas
            lemmas.add(token.get(LemmaAnnotation.class));
        }
    }
    return lemmas;
}




public static void main(String[] args) {

    System.out.println("Starting Stanford Lemmatizer");
   String text = "How could you be seeing into my eyes like open doors? \n"+
            "You led me down into my core where I've became so numb \n"+
            "Without a soul my spirit's sleeping somewhere cold \n"+
            "Until you find it there and led it back home \n"+
            "You woke me up inside \n"+
            "Called my name and saved me from the dark \n"+
            "You have bidden my blood and it ran \n"+
            "Before I would become undone \n"+
            "You saved me from the nothing I've almost become \n"+
            "You were bringing me to life \n"+
            "Now that I knew what I'm without \n"+
            "You can've just left me \n"+
            "You breathed into me and made me real \n"+
            "Frozen inside without your touch \n"+
            "Without your love, darling \n"+
            "Only you are the life among the dead \n"+
            "I've been living a lie, there's nothing inside \n"+
            "You were bringing me to life.";

    StanfordLemmatizer slem = new StanfordLemmatizer();
    System.out.println(slem.lemmatize(text));

Answer 1

如果你唯一需要的是词形还原，那么使用简单的句子会更好。

import edu.stanford.nlp.simple.Sentence;

public List<String> getLemmasList(String text) {
  Sentence sentence = new Sentence(text);
  return lemmas = sentence.lemmas();
}

你也可以通过以下方式获得单词part_of_speech：

 sentence.word(i);
 sentence.posTag(i);

Answer 2

你可能不会对你的字符串进行标记，并让它完全被词形化。

如果我理解正确，您需要删除您已经从属性中完成的这两个步骤。

tokenize, ssplit,

虽然，如果你已经执行了这些步骤，那么让他们留下来真的不会有任何伤害。单个字符串无法再次拆分。

如果你有一个字符串列表，你可以单独循环它们并在一个单词和句子上调用lemmatize方法。注意：您可以轻松编辑方法以从列表中返回唯一的引理字符串（或尝试从方法中删除列表）

Answer 3

您需要将项目中的语言模型作为库包含在内。该文件可在以下链接中找到，＆＃34; http://stanfordnlp.github.io/CoreNLP/＆＃34; 对于英文模型，文件名是＆＃34; stanford-english-corenlp-models-current.jar＆＃34;。还提供多种语言，包括中国，德国，阿拉伯语等。

Answer 4

我试图找到使用斯坦福核心NLP中的新更改进行词形化的方法，但是当前答案没有随CoreDocument的新用法而更新。我能够弄清楚-现在进行lemmatize，需要完成以下工作：

$stmt->execute(['question' => $question, 'beggining' => (int)trim($beggining)]);

使用StanfordCoreNLP的词形还原

4 个答案: