Question

我正在尝试从文件系统索引大量数据，数据大小为几TB。我正在使用FileListEntityProcessor和TikaEntityProcessor。

数据-config.xml中

<dataConfig>
    <dataSource name="bin" type="BinFileDataSource" />
    <document>
        <entity name="f" dataSource="null" rootEntity="true"
            processor="FileListEntityProcessor" transformer="TemplateTransformer"
            baseDir="//mathworks/devel/bat/A/logs/66048/"
            fileName=".*\.*" onError="skip" recursive="true">

            <field column="fileAbsolutePath" name="path" />
            <field column="fileSize" name="size"/>
            <field column="fileLastModified" name="lastmodified" />

            <entity name="file" dataSource="bin" processor="TikaEntityProcessor" url="${f.fileAbsolutePath}" format="text" onError="skip"
           rootEntity="true">
                <field column="text" name="text"/>   
            </entity>
        </entity>
    </document>
</dataConfig>

我正在使用-Xms1024m -Xmx10240m来说明jetty服务器，但我仍然不断获得Java堆异常，尽管我的机器非常好。索引数据库时遇到了同样的问题，但我的问题是用“batchSize = -1”解决的。我可以在这做些什么吗？或者有什么东西可以从文件系统中索引数TB的数据。

编辑：我正在尝试索引所有文本文件的日志文件。我没有看到任何特定文档未通过该过程。一个半小时后，它只显示异常。

INFO: [tika] webapp=null path=null params={event=newSearcher&q=static+newSearcher+warming+query+from+solrconfig.xml} hits=872 status=0 QTime=1
Oct 8, 2013 6:04:27 PM org.apache.solr.core.QuerySenderListener newSearcher
INFO: QuerySenderListener done.
Oct 8, 2013 6:04:27 PM org.apache.solr.core.SolrCore registerSearcher
INFO: [tika] Registered new searcher Searcher@15cab4c0 main
Oct 8, 2013 6:04:27 PM org.apache.solr.search.SolrIndexSearcher close
INFO: Closing Searcher@67071180 main
        fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=3,evictions=0,size=3,warmupTime=5,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=10,evictions=0,size=10,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
Oct 8, 2013 6:04:33 PM org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {deleteByQuery=*:*,add=[(null), (null), (null), (null), (null), (null), (null), (null), ... (20157 adds)]} 0 4
Oct 8, 2013 6:04:33 PM org.apache.solr.common.SolrException log
SEVERE: Full Import failed:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:264)
        at org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:375)
        at org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:445)
        at org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:426)
Caused by: java.lang.RuntimeException: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:621)
        at org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:327)
        at org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:225)
        ... 3 more
Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:759)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:619)
        ... 5 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Unknown Source)
        at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
        at java.lang.AbstractStringBuilder.append(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at org.apache.solr.common.SolrInputField.toString(SolrInputField.java:200)
        at java.lang.String.valueOf(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at java.util.AbstractMap.toString(Unknown Source)
        at java.lang.String.valueOf(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at org.apache.solr.common.SolrInputDocument.toString(SolrInputDocument.java:182)
        at java.lang.String.valueOf(Unknown Source)
        at java.lang.StringBuilder.append(Unknown Source)
        at org.apache.solr.handler.dataimport.SolrWriter.upload(SolrWriter.java:68)
        at org.apache.solr.handler.dataimport.DataImportHandler$1.upload(DataImportHandler.java:293)
        at org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:723)
        ... 6 more

Oct 8, 2013 6:04:33 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: start rollback
Oct 8, 2013 6:04:33 PM org.apache.solr.update.DirectUpdateHandler2 rollback
INFO: end_rollback

感谢adcvance。 Prerna

FileListEntityProcessor + java堆异常

0 个答案: