Solr Error实例化类:'org.apache.lucene.analysis.miscellaneous.TruncateTokenFilterFactory'

时间:2017-07-19 21:55:51

标签: java solr lucene wikipedia

我正在将wikipedia转储索引到solr-6.6.0。虽然有这样的错误 引起:

java.lang.IllegalArgumentException: Document contains at least one immense term in field="text" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[10, 10, 70, 111, 117, 110, 100, 32, 116, 104, 105, 115, 32, 111, 110, 32, 116, 104, 101, 32, 119, 101, 98, 32, 104, 111, 112, 101, 32, 116]...', original message: bytes can be at most 32766 in length; got 43225
    at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:671)
    at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:344)
    at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:300)
    at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:234)
    at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:450)
    at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1475)
    at org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
    at org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:163)
    ... 54 more
Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got 43225
    at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:284)

所以我在核心中修改了架构并添加了这一行。

<filter class="solr.TruncateTokenFilterFactory" prefixLength="32700"/>

但是当我重新启动solr服务器时,它总是给出错误通知:     [schema.xml]分析器/过滤器的插件初始化失败:实例化类的错误:'org.apache.lucene.analysis.miscellaneous.TruncateTokenFilterFactory'

我是否设置了值32700错了?最大长度是多少? 这是我的架构文件:

<?xml version="1.0" encoding="UTF-8" ?>
<schema version="1.5" name="wikipedia">

    <fieldType name="text_wiki" class="solr.TextField" positionIncrementGap="100">
        <analyzer type="index">
            <tokenizer class="solr.WikipediaTokenizerFactory"/>
            <filter class="solr.StopFilterFactory"
                    ignoreCase="true"
                    words="lang/stopwords_en.txt"
                    />
            <filter class="solr.LowerCaseFilterFactory"/>
            <filter class="solr.EnglishPossessiveFilterFactory"/>
            <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
            <!-- Optionally you may want to use this less aggressive stemmer instead of PorterStemFilterFactory:
            <filter class="indexer.EnglishMinimalStemFilterFactory"/> -->
            <filter class="solr.PorterStemFilterFactory"/>
            <filter class="solr.LengthFilterFactory" min="2" max="32700" />
        </analyzer>
        <analyzer type="query">
            <tokenizer class="solr.StandardTokenizerFactory"/>
            <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
            <filter class="solr.StopFilterFactory"
                    ignoreCase="true"
                    words="lang/stopwords_en.txt"
                    />
            <filter class="solr.LowerCaseFilterFactory"/>
            <filter class="solr.EnglishPossessiveFilterFactory"/>
            <filter class="solr.KeywordMarkerFilterFactory" protected="protwords.txt"/>
            <!-- Optionally you may want to use this less aggressive stemmer instead of PorterStemFilterFactory:
            <filter class="indexer.EnglishMinimalStemFilterFactory"/> -->
            <filter class="solr.PorterStemFilterFactory"/>
        </analyzer>
    </fieldType>

    <fieldType name="string" class="solr.StrField"/>
    <fieldType name="date" class="solr.TrieDateField" precisionStep="0" positionIncrementGap="0"/>
    <fieldType name="int" class="solr.TrieIntField" precisionStep="0" positionIncrementGap="0"/>
    <fieldType name="float" class="solr.TrieFloatField" precisionStep="0" positionIncrementGap="0"/>
    <fieldType name="long" class="solr.TrieLongField" precisionStep="0" positionIncrementGap="0"/>
    <fieldType name="double" class="solr.TrieDoubleField" precisionStep="0" positionIncrementGap="0"/>

    <field name="id" type="string" indexed="true" stored="true" required="true"/>
    <field name="_version_" type="long" indexed="true" stored="true"/>
    <field name="TITLE" type="text_wiki" indexed="true" stored="true" termVectors="true" termPositions="true" termOffsets="true" />
    <field name="REVISION_TEXT" type="text_wiki" indexed="true" stored="true" multiValued="true" termVectors="true" termPositions="true" termOffsets="true" />
    <field name="REVISION_TIMESTAMP" type="date" indexed="true" stored="true" multiValued="true" />
    <field name="CONTRIBUTOR_ID" type="int" indexed="true" stored="true" multiValued="true" />
    <field name="CONTRIBUTOR_USERNAME" type="string" indexed="true" docValues="true" stored="true" multiValued="true" />

    <dynamicField name="*" type="string" indexed="true" stored="true" multiValued="true"/>
    <uniqueKey>id</uniqueKey>
</schema>

0 个答案:

没有答案