Question

我指的是下面给出的Cloudera搜索索引文档 -

https://www.cloudera.com/documentation/enterprise/5-9-x/topics/search_data_index_prepare.html https://www.cloudera.com/documentation/enterprise/5-9-x/topics/search_batch_index_use_mapreduce.html

我根据csv格式的数据集准备了集合，模式文件和morphline文件。

id  jobtitle    jobdescription  city    state   classification  salary
1 Senior Android Developer  complex problem solving New Hope    PA  it  94036
2 Mobile Solutions Developer complex problem solving    Glen Allen  VA  it 60726

我使用的MRIT命令是：

sudo -u hdfs hadoop \
--config /etc/hadoop/conf.cloudera.yarn \
jar /opt/cloudera/parcels/CDH/lib/solr/contrib/mr/search-mr-*-job.jar 
org.apache.solr.hadoop.MapReduceIndexerTool \
-D 'mapred.child.java.opts=-Xmx500m' \
--log4j /opt/cloudera/parcels/CDH/share/doc/search-
1.0.0+cdh5.8.3+0/examples/solr-nrt/log4j.properties \
--morphline-file $HOME/jobs.conf \
--output-dir NN:8020/user/$USER/outdir \
--zk-host localhost/solr  --collection jobs\
--go-live \
NN:8020/user/$USER/indir

以下是我的架构文件 -

<?xml version="1.0" encoding="UTF-8" ?>
<schema name="example" version="1.5">
<fields>

    <!-- Posts -->
    <field name="id" type="string" indexed="true" stored="true" 
    required="true"/>
    <field name="jobtitle" type="text_general" indexed="true" 
    stored="true"/>
    <field name="jobdescription" type="text_general" indexed="true" 
    stored="true" termVectors="true"/>
    <field name="classification" type="splitOnPeriod" indexed="true" 
    stored="true"/>
    <field name="city" type="text_general" indexed="true" stored="true"/>
    <field name="state" type="text_general" indexed="true" stored="true"/>
    <field name="salary" type="int" indexed="true" stored="true"/>
    <field name="_version_" type="long" indexed="true" stored="true"/> 
    <field name="content" type="text_general" indexed="true" stored="true" 
    multiValued="true"/> 
    <field name="text" type="text_general" indexed="false" stored="true" 
    multiValued="true"/>   

    <copyField source="jobtitle" dest="content" />   
    <copyField source="jobdescription" dest="content" />  
</fields>

<types>
    <fieldType name="string" class="solr.StrField" sortMissingLast="true" />
    <fieldType name="int" class="solr.TrieIntField" precisionStep="0" 
    positionIncrementGap="0"/>
    <fieldType name="long" class="solr.TrieLongField" precisionStep="0"                 
    positionIncrementGap="0"/>
    <fieldType name="date" class="solr.TrieDateField" precisionStep="0" 
    positionIncrementGap="0"/>

    <fieldType name="text_general" class="solr.TextField" 
    positionIncrementGap="100">
        <analyzer>
            <tokenizer class="solr.StandardTokenizerFactory"/>
            <filter class="solr.LowerCaseFilterFactory"/>
        </analyzer>
    </fieldType>

    <fieldType name="splitOnPeriod" class="solr.TextField" 
    positionIncrementGap="100">
         <analyzer>
            <tokenizer class="solr.PatternTokenizerFactory" pattern="\." />
            <filter class="solr.LowerCaseFilterFactory"/>
        </analyzer>
    </fieldType>        
</types>

<uniqueKey>id</uniqueKey>

</schema>

我做了一次干运行并且它运行但是上线后我总是得到MAX_ARRAY_LENGTH错误。

1554 [main] INFO  org.apache.solr.hadoop.MapReduceIndexerTool  - Indexing 1 
files using 1 real mappers into 2 reducers
Error: MAX_ARRAY_LENGTH

错误似乎在映射阶段。

1686 [main] ERROR org.apache.hadoop.mapred.YarnChild  - Error running child 
: java.lang.NoSuchFieldError: MAX_ARRAY_LENGTH
    at org.apache.lucene.codecs.memory.DirectDocValuesFormat.<clinit>
 (DirectDocValuesFormat.java:58)

请帮我解决这个问题。

Answer 1

当您的环境中未正确安装某些内容且org / apache / lucene / util / ArrayUtil.class中的max_array_length属性设置为1时，通常会发生此错误。您可以升级CDH以消除错误，也可以将该java类中变量的堆大小增加到2或更大。我在不同的环境中尝试了相同的MRIT命令，它工作正常。

参考资料 - https://lucene.apache.org/core/4_7_0/core/org/apache/lucene/util/ArrayUtil.html

在Solr中索引数据时如何解决MAX_ARRAY_LENGTH错误？

1 个答案: