Question

我在Ubuntu 14.04上使用Python3，并且在67条原始文章的语料库中运行Stanford POSTagger，编辑的python脚本如下：

from nltk.tag.stanford import POSTagger

with open('the_file.txt','r') as file:
    G=file.readlines()

stan=[]

english_postagger = POSTagger('models/english-bidirectional-distsim.tagger', 'stanford-postagger.jar')

for line in g:
    stan.append(english_postagger.tag(tokenize_fast(line)))

经过几次迭代后，我得到以下错误：

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at edu.stanford.nlp.sequences.ExactBestSequenceFinder.bestSequence(ExactBestSequenceFinder.java:109)
at edu.stanford.nlp.sequences.ExactBestSequenceFinder.bestSequence(ExactBestSequenceFinder.java:31)
at edu.stanford.nlp.tagger.maxent.TestSentence.runTagInference(TestSentence.java:322)
at edu.stanford.nlp.tagger.maxent.TestSentence.testTagInference(TestSentence.java:312)
at edu.stanford.nlp.tagger.maxent.TestSentence.tagSentence(TestSentence.java:135)
at edu.stanford.nlp.tagger.maxent.MaxentTagger.tagSentence(MaxentTagger.java:998)
at edu.stanford.nlp.tagger.maxent.MaxentTagger.tagCoreLabelsOrHasWords(MaxentTagger.java:1788)
at edu.stanford.nlp.tagger.maxent.MaxentTagger.tagAndOutputSentence(MaxentTagger.java:1798)
at edu.stanford.nlp.tagger.maxent.MaxentTagger.runTagger(MaxentTagger.java:1709)
at edu.stanford.nlp.tagger.maxent.MaxentTagger.runTagger(MaxentTagger.java:1770)
at edu.stanford.nlp.tagger.maxent.MaxentTagger.runTagger(MaxentTagger.java:1543)
at edu.stanford.nlp.tagger.maxent.MaxentTagger.runTagger(MaxentTagger.java:1499)
at edu.stanford.nlp.tagger.maxent.MaxentTagger.main(MaxentTagger.java:1842)

我还从命令行运行stanford postagger：

java -mx300m -classpath stanford-postagger.jar   edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/wsj-0-18-bidirectional-distsim.tagger -textFile sample-input.txt > sample-tagged.txt

有类似的错误。我甚至通过了Java 2 GB的内存，但仍然没有运气。

非常欢迎任何想法/想法或hacky类型的解决方案！

很好地发现了@nsanglar，所以我试过了：

java -Xmx2g -classpath stanford-postagger.jar   edu.stanford.nlp.tagger.maxent.MaxentTagger -model models/wsj-0-18-bidirectional-distsim.tagger -textFile raw_text.txt > sample-tagged.txt

我收到错误日志消息，标题如下：

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 283639808 bytes for committing   reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#     Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.

#  Out of Memory Error (os_linux.cpp:2798), pid=25677, tid=140571167794944

# JRE version: OpenJDK Runtime Environment (7.0_65-b32) (build 1.7.0_65-b32)
# Java VM: OpenJDK 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 2.5.2
# Distribution: Ubuntu 14.04 LTS, package 7u65-2.5.2-3~14.04
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try  "ulimit -c unlimited" before starting Java again

Answer 1

嗯，事实证明这是一个RAM问题，我根本没有足够的内存来执行命令。从服务器上运行标记器可以解决问题。

Answer 2

你应该使用-Xmx1024m。我认为你打错了，因为目前你正在使用-mx：）

Answer 3

在python中：

nltk.internals.config_java(options='-Xmx3024m')

Stanford POS Tagger，Java堆空间内存

3 个答案: