索引恢复后Elasticsearch不响应

时间:2014-06-22 08:21:37

标签: elasticsearch

ES在丢失索引后没有响应任何请求(原因不明)。服务器重新启动ES尝试恢复索引后,但只要它读取整个索引(仅约200mb)ES停止响应。我看到的最后一个错误是SearchPhaseExecutionException[Failed to execute phase [query_fetch], all shards failed]。我在单节点虚拟服务器上使用ES。索引只有一个碎片,大约有3个文件(200mb)。

如何恢复此索引?

这是ES日志

[2014-06-21 18:43:15,337][WARN ][bootstrap                ] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line
[2014-06-21 18:43:15,554][WARN ][common.jna               ] Unknown mlockall error 0
[2014-06-21 18:43:15,759][INFO ][node                     ] [Crimson Cowl] version[1.1.0], pid[1031], build[2181e11/2014-03-25T15:59:51Z]
[2014-06-21 18:43:15,759][INFO ][node                     ] [Crimson Cowl] initializing ...
[2014-06-21 18:43:15,881][INFO ][plugins                  ] [Crimson Cowl] loaded [], sites [head]
[2014-06-21 18:43:21,957][INFO ][node                     ] [Crimson Cowl] initialized
[2014-06-21 18:43:21,958][INFO ][node                     ] [Crimson Cowl] starting ...
[2014-06-21 18:43:22,275][INFO ][transport                ] [Crimson Cowl] bound_address {inet[/10.0.0.13:9300]}, publish_address {inet[/10.0.0.13:9300]}
[2014-06-21 18:43:25,385][INFO ][cluster.service          ] [Crimson Cowl] new_master [Crimson Cowl][UJNl8hGgRzeFo-DQ3vk2nA][esubuntu][inet[/10.0.0.13:9300]], reason: zen-disco-join (elected_as_master)
[2014-06-21 18:43:25,438][INFO ][discovery                ] [Crimson Cowl] elasticsearch/UJNl8hGgRzeFo-DQ3vk2nA
[2014-06-21 18:43:25,476][INFO ][http                     ] [Crimson Cowl] bound_address {inet[/10.0.0.13:9200]}, publish_address {inet[/10.0.0.13:9200]}
[2014-06-21 18:43:26,348][INFO ][gateway                  ] [Crimson Cowl] recovered [2] indices into cluster_state
[2014-06-21 18:43:26,349][INFO ][node                     ] [Crimson Cowl] started

删除同一节点上的另一个索引后,ES响应请求,但无法恢复索引。这是日志

[2014-06-22 08:00:06,651][WARN ][bootstrap                ] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line
[2014-06-22 08:00:06,699][WARN ][common.jna               ] Unknown mlockall error 0
[2014-06-22 08:00:06,774][INFO ][node                     ] [Baron Macabre] version[1.1.0], pid[2035], build[2181e11/2014-03-25T15:59:51Z]
[2014-06-22 08:00:06,774][INFO ][node                     ] [Baron Macabre] initializing ...
[2014-06-22 08:00:06,779][INFO ][plugins                  ] [Baron Macabre] loaded [], sites [head]
[2014-06-22 08:00:08,766][INFO ][node                     ] [Baron Macabre] initialized
[2014-06-22 08:00:08,767][INFO ][node                     ] [Baron Macabre] starting ...
[2014-06-22 08:00:08,824][INFO ][transport                ] [Baron Macabre] bound_address {inet[/10.0.0.3:9300]}, publish_address {inet[/10.0.0.3:9300]}
[2014-06-22 08:00:11,890][INFO ][cluster.service          ] [Baron Macabre] new_master [Baron Macabre][eWDP4ZSXSGuASJLJ2an1nQ][esubuntu][inet[/10.0.0.3:9300]], reason: zen-disco-join (elected_as_master)
[2014-06-22 08:00:11,975][INFO ][discovery                ] [Baron Macabre] elasticsearch/eWDP4ZSXSGuASJLJ2an1nQ
[2014-06-22 08:00:12,000][INFO ][http                     ] [Baron Macabre] bound_address {inet[/10.0.0.3:9200]}, publish_address {inet[/10.0.0.3:9200]}
[2014-06-22 08:00:12,645][INFO ][gateway                  ] [Baron Macabre] recovered [1] indices into cluster_state
[2014-06-22 08:00:12,647][INFO ][node                     ] [Baron Macabre] started
[2014-06-22 08:05:01,284][WARN ][index.engine.internal    ] [Baron Macabre] [wordstat][0] failed engine
java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.index.ParallelPostingsArray.<init>(ParallelPostingsArray.java:35)
        at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.<init>(FreqProxTermsWriterPerField.java:254)
        at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.newInstance(FreqProxTermsWriterPerField.java:279)
        at org.apache.lucene.index.ParallelPostingsArray.grow(ParallelPostingsArray.java:48)
        at org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray.grow(TermsHashPerField.java:307)
        at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:324)
        at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:185)
        at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:171)
        at org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:248)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:253)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:453)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1529)
        at org.elasticsearch.index.engine.internal.InternalEngine.innerIndex(InternalEngine.java:532)
        at org.elasticsearch.index.engine.internal.InternalEngine.index(InternalEngine.java:470)
        at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:744)
        at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:228)
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:197)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
[2014-06-22 08:05:02,168][WARN ][cluster.action.shard     ] [Baron Macabre] [wordstat][0] sending failed shard for [wordstat][0], node[eWDP4ZSXSGuASJLJ2an1nQ], [P], s[INITIALIZING], indexUUID [LC3LMLxgS3CkkG_pvfTeSg], reason [engine failure, message [OutOfMemoryError[Java heap space]]]
[2014-06-22 08:05:02,169][WARN ][cluster.action.shard     ] [Baron Macabre] [wordstat][0] received shard failed for [wordstat][0], node[eWDP4ZSXSGuASJLJ2an1nQ], [P], s[INITIALIZING], indexUUID [LC3LMLxgS3CkkG_pvfTeSg], reason [engine failure, message [OutOfMemoryError[Java heap space]]]
[2014-06-22 08:53:22,253][INFO ][node                     ] [Baron Macabre] stopping ...
[2014-06-22 08:53:22,267][INFO ][node                     ] [Baron Macabre] stopped
[2014-06-22 08:53:22,267][INFO ][node                     ] [Baron Macabre] closing ...
[2014-06-22 08:53:22,272][INFO ][node                     ] [Baron Macabre] closed
[2014-06-22 08:53:23,667][WARN ][bootstrap                ] jvm uses the client vm, make sure to run `java` with the server vm for best performance by adding `-server` to the command line
[2014-06-22 08:53:23,708][WARN ][common.jna               ] Unknown mlockall error 0
[2014-06-22 08:53:23,777][INFO ][node                     ] [Living Totem] version[1.1.0], pid[2137], build[2181e11/2014-03-25T15:59:51Z]
[2014-06-22 08:53:23,777][INFO ][node                     ] [Living Totem] initializing ...
[2014-06-22 08:53:23,781][INFO ][plugins                  ] [Living Totem] loaded [], sites [head]
[2014-06-22 08:53:25,828][INFO ][node                     ] [Living Totem] initialized
[2014-06-22 08:53:25,828][INFO ][node                     ] [Living Totem] starting ...
[2014-06-22 08:53:25,885][INFO ][transport                ] [Living Totem] bound_address {inet[/10.0.0.3:9300]}, publish_address {inet[/10.0.0.3:9300]}
[2014-06-22 08:53:28,913][INFO ][cluster.service          ] [Living Totem] new_master [Living Totem][D-eoRm7fSrCU_dTw_NQipA][esubuntu][inet[/10.0.0.3:9300]], reason: zen-disco-join (elected_as_master)
[2014-06-22 08:53:28,939][INFO ][discovery                ] [Living Totem] elasticsearch/D-eoRm7fSrCU_dTw_NQipA
[2014-06-22 08:53:28,964][INFO ][http                     ] [Living Totem] bound_address {inet[/10.0.0.3:9200]}, publish_address {inet[/10.0.0.3:9200]}
[2014-06-22 08:53:29,433][INFO ][gateway                  ] [Living Totem] recovered [1] indices into cluster_state
[2014-06-22 08:53:29,433][INFO ][node                     ] [Living Totem] started
[2014-06-22 08:58:05,268][WARN ][index.engine.internal    ] [Living Totem] [wordstat][0] failed engine
java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.<init>(FreqProxTermsWriterPerField.java:261)
        at org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray.newInstance(FreqProxTermsWriterPerField.java:279)
        at org.apache.lucene.index.ParallelPostingsArray.grow(ParallelPostingsArray.java:48)
        at org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray.grow(TermsHashPerField.java:307)
        at org.apache.lucene.util.BytesRefHash.add(BytesRefHash.java:324)
        at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:185)
        at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:171)
        at org.apache.lucene.index.DocFieldProcessor.processDocument(DocFieldProcessor.java:248)
        at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:253)
        at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:453)
        at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1529)
        at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1199)
        at org.elasticsearch.index.engine.internal.InternalEngine.innerIndex(InternalEngine.java:523)
        at org.elasticsearch.index.engine.internal.InternalEngine.index(InternalEngine.java:470)
        at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:744)
        at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:228)
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:197)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
[2014-06-22 08:58:06,046][WARN ][cluster.action.shard     ] [Living Totem] [wordstat][0] sending failed shard for [wordstat][0], node[D-eoRm7fSrCU_dTw_NQipA], [P], s[INITIALIZING], indexUUID [LC3LMLxgS3CkkG_pvfTeSg], reason [engine failure, message [OutOfMemoryError[Java heap space]]]
[2014-06-22 08:58:06,047][WARN ][cluster.action.shard     ] [Living Totem] [wordstat][0] received shard failed for [wordstat][0], node[D-eoRm7fSrCU_dTw_NQipA], [P], s[INITIALIZING], indexUUID [LC3LMLxgS3CkkG_pvfTeSg], reason [engine failure, message [OutOfMemoryError[Java heap space]]]

2 个答案:

答案 0 :(得分:0)

为了恢复您的Elasticsearch集群,您需要为堆分配更多内存。当你在一个相当小的实例上运行时,这可能有点挑战,但这是你需要做的事情:

  1. 更改配置以将更多内存分配给堆。不
    清除您当前的设置,但有几种方法 提升这一点 - 最简单的方法是设置环境变量
    ES_HEAP_SIZE。我从1GB开始,尝试一下,然后将其提升 小的增量,因为你已经接近你的极限 使用1.6GB内存实例。或者你也可以 用于启动Elasticsearch的文件的更改取决于 你如何安装它们,但应该在bin目录中 在Elasticsearch主目录下。对于Linux 安装文件为elasticsearchelasticsearch.in.sh
  2. 转到更大的实例。这将更容易恢复 在具有更多内存的系统上 - 如果上述步骤不起作用, 您可以将所有文件复制到另一个更大的实例并尝试使用 上面的步骤再次使用更大的堆大小。

答案 1 :(得分:-3)

这里发生的事情是您的服务器已经过载。可能有一个坏的部门。您需要做的是删除现有索引并重新索引它们。

在Linux上,

  • Elasticsearch临时文件保存在usr/local/var/elasticsearch/
  • 删除此文件夹,然后重新填充索引