Question

我正在使用Solr 6.5.1

我在合并3.5TB索引时遇到了以下错误。导致这种情况的原因是什么？

我正在使用Solr的IndexMergeTool，来自以下命令：

java -classpath lucene-core-6.5.1.jar;lucene-misc-6.5.1.jar org.apache.lucene.misc.IndexMergeTool

堆大小为32GB。两个核心中有超过2000万个文档。这些3.5TB只有1个主机，1个节点和1个分片。在达到此错误之前，合并已运行超过48小时。

Exception in thread "main" Exception in thread "Lucene Merge Thread #8" java.io.
IOException: background merge hit exception: _6f(6.5.1):C7256757 _6e(6.5.1):C646
2072 _6d(6.5.1):C3750777 _6c(6.5.1):C2243594 _6b(6.5.1):C1015431 _6a(6.5.1):C105
0220 _69(6.5.1):c273879 _28(6.4.1):c79011/84:delGen=84 _26(6.4.1):c44960/8149:de
  lGen=100 _29(6.4.1):c73855/68:delGen=68 _5(6.4.1):C46672/31:delGen=31 _68(6.5.1)
:c66 into _6g [maxNumSegments=1]
        at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1931)

        at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1871)

        at org.apache.lucene.misc.IndexMergeTool.main(IndexMergeTool.java:57)
Caused by: java.io.IOException: The requested operation could not be completed d
ue to a file system limitation
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.write(Unknown Source)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
        at sun.nio.ch.IOUtil.write(Unknown Source)
        at sun.nio.ch.FileChannelImpl.write(Unknown Source)
        at java.nio.channels.Channels.writeFullyImpl(Unknown Source)
        at java.nio.channels.Channels.writeFully(Unknown Source)
        at java.nio.channels.Channels.access$000(Unknown Source)
        at java.nio.channels.Channels$1.write(Unknown Source)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory
.java:419)
        at java.util.zip.CheckedOutputStream.write(Unknown Source)
        at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
        at java.io.BufferedOutputStream.write(Unknown Source)
        at org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStre
amIndexOutput.java:53)
        at org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimited
IndexOutput.java:73)
        at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
        at org.apache.lucene.codecs.lucene50.ForUtil.writeBlock(ForUtil.java:175
)
        at org.apache.lucene.codecs.lucene50.Lucene50PostingsWriter.addPosition(
Lucene50PostingsWriter.java:286)
        at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPosting
sWriterBase.java:156)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.w
rite(BlockTreeTermsWriter.java:866)
        at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTr
eeTermsWriter.java:344)
        at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105
)
        at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter
.merge(PerFieldPostingsFormat.java:164)
        at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:2
16)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4353
)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3928)
        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMe
rgeScheduler.java:624)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Conc
urrentMergeScheduler.java:661)
org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: The req
uested operation could not be completed due to a file system limitation
        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException
(ConcurrentMergeScheduler.java:703)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(Conc
urrentMergeScheduler.java:683)
Caused by: java.io.IOException: The requested operation could not be completed d
ue to a file system limitation
        at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at sun.nio.ch.FileDispatcherImpl.write(Unknown Source)
        at sun.nio.ch.IOUtil.writeFromNativeBuffer(Unknown Source)
        at sun.nio.ch.IOUtil.write(Unknown Source)
        at sun.nio.ch.FileChannelImpl.write(Unknown Source)
        at java.nio.channels.Channels.writeFullyImpl(Unknown Source)
        at java.nio.channels.Channels.writeFully(Unknown Source)
        at java.nio.channels.Channels.access$000(Unknown Source)
        at java.nio.channels.Channels$1.write(Unknown Source)

的问候，
埃德

Answer 1

这是Windows主机吗？

合并操作遇到了文件系统的一些问题，因为索引是3.5TB，有些文件可能非常大，所以根据Windows的版本，你可能会遇到其中一个限制。

你可以做的事情：

在solrconfig.xml中确保这是假的（所以整个索引放在多个文件中，而不是一个文件中）
```
<useCompoundFile>false</useCompoundFile>
```
合并为2或3段（如果您合并为1）：

＆安培; maxSegments = 2
尝试对磁盘进行碎片整理
升级到更高版本的Windows Server，每个新版本的文件大小限制等都会更大
转移到linux

在Solr中合并索引期间出错

1 个答案: