我们在AWS上运行的Elasticsearch集群遇到了严重问题，导致无法对该集群进行更改。
我们的设置非常正常，除了我们白天通过添加/删除实例并根据需要增加/减少每个索引的副本数来扩展群集。 在问题发生之前大约一周，我们每次扩展集群时就开始一次添加两个Elasticsearch节点。
紧急群集任务阻止了所有其他更改群集状态的任务（节点连接/叶子，索引设置更改...）。基本上，它会迫使群集进入只读状态，并且由于节点连接任务也会被阻塞，因此无法放大或缩小群集。
GET _cluster/pending_tasks
{
"tasks": [
{
"insert_order": 4903913,
"priority": "URGENT",
"source": "shard-started StartedShardEntry{shardId [[hsearch_place-1541119291-1541107201707][0]], allocationId [HDkSthfcRyq6-1zdO0N1NA], message [after peer recovery]}",
"executing": true,
"time_in_queue_millis": 351128747,
"time_in_queue": "4d"
},
{
"insert_order": 4904859,
"priority": "IMMEDIATE",
"source": "zen-disco-node-left({search10}{UpapDMtiQQGsHt18XU_Obg}{hY9KJfUzRYCztq4N3fCpxw}{10.77.8.13}{10.77.8.13:9300}{ml.machine_memory=32899215360, rack=eu-west-1a, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}), reason(left)",
"executing": false,
"time_in_queue_millis": 308604643,
"time_in_queue": "3.5d"
},
....
}
[2018-11-02T06:07:00,964][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [search01] updating number_of_replicas to [2] for indices [hsearch_place-1541119291-1541107201707]
[2018-11-02T06:07:00,989][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [search01] updating number_of_replicas to [2] for indices [hsearch_product-1541121966-1541107201707]
[2018-11-02T06:07:01,019][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [search01] updating number_of_replicas to [2] for indices [hsearch_autocomplete-1541120879-1541107201707]
[2018-11-02T06:09:29,139][INFO ][o.e.c.s.MasterService ] [search01] zen-disco-node-join, reason: added {{search06}{O3BIQMIFR2-MIOrR_8Rswg}{5SmazoiiQVuH8LiuZbm5dQ}{10.77.8.12}{10.77.8.12:9300}{ml.machine_memory=32899215360, rack=eu-west-1a, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}
[2018-11-02T06:09:32,206][INFO ][o.e.c.s.ClusterApplierService] [search01] added {{search06}{O3BIQMIFR2-MIOrR_8Rswg}{5SmazoiiQVuH8LiuZbm5dQ}{10.77.8.12}{10.77.8.12:9300}{ml.machine_memory=32899215360, rack=eu-west-1a, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {search01}{ACzv-eLuR8OGKGaVfzrZBQ}{w7ppkDQAQKe2HzQeRJ1Hjw}{10.77.8.10}{10.77.8.10:9300}{ml.machine_memory=8002396160, rack=eu-west-1a, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [666678] source [zen-disco-node-join]])
[2018-11-02T06:10:35,393][INFO ][o.e.c.m.MetaDataMappingService] [search01] [hsearch_company-1541117051-1541107201707/xqp3YzGMRuqTWNYmOLwLbw] update_mapping [master]
[2018-11-02T06:10:49,685][INFO ][o.e.c.s.MasterService ] [search01] zen-disco-node-join, reason: added {{search05}{bD5y0cMSTrej0HTTsm2Klg}{Do-92dxsQhygLBFyYyXU-w}{10.77.8.11}{10.77.8.11:9300}{ml.machine_memory=32899215360, rack=eu-west-1a, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}
[2018-11-02T06:10:52,709][INFO ][o.e.c.s.ClusterApplierService] [search01] added {{search05}{bD5y0cMSTrej0HTTsm2Klg}{Do-92dxsQhygLBFyYyXU-w}{10.77.8.11}{10.77.8.11:9300}{ml.machine_memory=32899215360, rack=eu-west-1a, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}, reason: apply cluster state (from master [master {search01}{ACzv-eLuR8OGKGaVfzrZBQ}{w7ppkDQAQKe2HzQeRJ1Hjw}{10.77.8.10}{10.77.8.10:9300}{ml.machine_memory=8002396160, rack=eu-west-1a, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true} committed version [666680] source [zen-disco-node-join]])
[2018-11-02T06:11:02,188][INFO ][o.e.c.m.MetaDataMappingService] [search01] [hsearch_person-1541122087-1541107201707/s7AxcsI8T1mxdEd1J6-jbw] update_mapping [master]
[2018-11-02T06:14:00,983][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [search01] updating number_of_replicas to [3] for indices [hsearch_person-1541122087-1541107201707]
[2018-11-02T06:18:26,704][INFO ][o.e.c.r.a.AllocationService] [search01] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[hsearch_person-1541122087-1541107201707][4]] ...]).
[2018-11-02T06:23:01,443][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [search01] updating number_of_replicas to [3] for indices [hsearch_person-1541122087-1541107201707]
[2018-11-02T06:23:01,503][INFO ][o.e.c.m.MetaDataUpdateSettingsService] [search01] updating number_of_replicas to [3] for indices [hsearch_company-1541117051-1541107201707]
[2018-11-02T06:23:01,652][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [search01] [hsearch_company-1541117051-1541107201707][1]: failed to list shard for shard_store on node [O3BIQMIFR2-MIOrR_8Rswg]
org.elasticsearch.action.FailedNodeException: Failed node [O3BIQMIFR2-MIOrR_8Rswg]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:237) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1095) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TcpTransport.lambda$handleException$34(TcpTransport.java:1510) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:135) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1508) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TcpTransport.handlerResponseError(TcpTransport.java:1500) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1430) [elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:64) [transport-netty4-6.3.0.jar:6.3.0]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
Caused by: org.elasticsearch.transport.RemoteTransportException: [search06][10.77.8.12:9300][internal:cluster/nodes/indices/shard/store[n]]
Caused by: org.elasticsearch.ElasticsearchException: Failed to list store metadata for shard [[hsearch_company-1541117051-1541107201707][1]]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:111) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:260) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:256) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:246) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:304) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1592) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_162]
... 1 more
Caused by: java.io.FileNotFoundException: no segments* file found in store(MMapDirectory@/data/elasticsearch/nodes/0/indices/xqp3YzGMRuqTWNYmOLwLbw/1/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@7961bae4): files: [recovery.tMp0C0AQSb-jFrqFryYFnA._16.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._16.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._16.si, recovery.tMp0C0AQSb-jFrqFryYFnA._16_5.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._1j.dii, recovery.tMp0C0AQSb-jFrqFryYFnA._1j.dim, recovery.tMp0C0AQSb-jFrqFryYFnA._1j.fdx, recovery.tMp0C0AQSb-jFrqFryYFnA._1j.fnm, recovery.tMp0C0AQSb-jFrqFryYFnA._1j.nvd, recovery.tMp0C0AQSb-jFrqFryYFnA._1j.nvm, recovery.tMp0C0AQSb-jFrqFryYFnA._1j.si, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_9.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_Lucene50_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_Lucene50_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_Lucene50_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_Lucene50_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_Lucene50_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_Lucene70_0.dvd, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_Lucene70_0.dvm, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_completion_0.cmp, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_completion_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_completion_0.lkp, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_completion_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_completion_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_completion_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._1j_completion_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._21.dii, recovery.tMp0C0AQSb-jFrqFryYFnA._21.dim, recovery.tMp0C0AQSb-jFrqFryYFnA._21.fdt, recovery.tMp0C0AQSb-jFrqFryYFnA._21.fdx, recovery.tMp0C0AQSb-jFrqFryYFnA._21.fnm, recovery.tMp0C0AQSb-jFrqFryYFnA._21.nvd, recovery.tMp0C0AQSb-jFrqFryYFnA._21.nvm, recovery.tMp0C0AQSb-jFrqFryYFnA._21.si, recovery.tMp0C0AQSb-jFrqFryYFnA._21_8.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._21_Lucene50_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._21_Lucene50_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._21_Lucene50_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._21_Lucene50_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._21_Lucene50_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._21_Lucene70_0.dvd, recovery.tMp0C0AQSb-jFrqFryYFnA._21_Lucene70_0.dvm, recovery.tMp0C0AQSb-jFrqFryYFnA._21_completion_0.cmp, recovery.tMp0C0AQSb-jFrqFryYFnA._21_completion_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._21_completion_0.lkp, recovery.tMp0C0AQSb-jFrqFryYFnA._21_completion_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._21_completion_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._21_completion_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._21_completion_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._2c.dii, recovery.tMp0C0AQSb-jFrqFryYFnA._2c.dim, recovery.tMp0C0AQSb-jFrqFryYFnA._2c.fdx, recovery.tMp0C0AQSb-jFrqFryYFnA._2c.fnm, recovery.tMp0C0AQSb-jFrqFryYFnA._2c.nvd, recovery.tMp0C0AQSb-jFrqFryYFnA._2c.nvm, recovery.tMp0C0AQSb-jFrqFryYFnA._2c.si, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_8.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_Lucene50_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_Lucene50_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_Lucene50_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_Lucene50_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_Lucene50_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_Lucene70_0.dvd, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_Lucene70_0.dvm, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_completion_0.cmp, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_completion_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_completion_0.lkp, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_completion_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_completion_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_completion_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._2c_completion_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._2l.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._2l.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._2l.si, recovery.tMp0C0AQSb-jFrqFryYFnA._2l_8.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._2w.dii, recovery.tMp0C0AQSb-jFrqFryYFnA._2w.dim, recovery.tMp0C0AQSb-jFrqFryYFnA._2w.fdx, recovery.tMp0C0AQSb-jFrqFryYFnA._2w.fnm, recovery.tMp0C0AQSb-jFrqFryYFnA._2w.nvd, recovery.tMp0C0AQSb-jFrqFryYFnA._2w.nvm, recovery.tMp0C0AQSb-jFrqFryYFnA._2w.si, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_Lucene50_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_Lucene50_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_Lucene50_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_Lucene50_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_Lucene50_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_Lucene70_0.dvd, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_Lucene70_0.dvm, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_a.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_completion_0.cmp, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_completion_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_completion_0.lkp, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_completion_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_completion_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_completion_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._2w_completion_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._2y.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._2y.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._2y.si, recovery.tMp0C0AQSb-jFrqFryYFnA._30.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._30.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._30.si, recovery.tMp0C0AQSb-jFrqFryYFnA._30_1.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._35.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._35.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._35.si, recovery.tMp0C0AQSb-jFrqFryYFnA._35_2.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._38.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._38.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._38.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3b.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3b.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3b.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3e.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3e.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3e.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3f.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3f.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3f.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3f_1.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._3g.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3g.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3g.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3h.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3h.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3h.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3i.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3i.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3i.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3j.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3j.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3j.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3j_1.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._3k.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3k.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3k.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3l.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3l.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3l.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3m.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3m.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3m.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3n.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3n.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3n.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3o.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3o.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3o.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3p.dii, recovery.tMp0C0AQSb-jFrqFryYFnA._3p.dim, recovery.tMp0C0AQSb-jFrqFryYFnA._3p.fdt, recovery.tMp0C0AQSb-jFrqFryYFnA._3p.fdx, recovery.tMp0C0AQSb-jFrqFryYFnA._3p.fnm, recovery.tMp0C0AQSb-jFrqFryYFnA._3p.nvd, recovery.tMp0C0AQSb-jFrqFryYFnA._3p.nvm, recovery.tMp0C0AQSb-jFrqFryYFnA._3p.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_1.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_Lucene50_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_Lucene50_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_Lucene50_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_Lucene50_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_Lucene50_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_Lucene70_0.dvd, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_Lucene70_0.dvm, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_completion_0.cmp, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_completion_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_completion_0.lkp, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_completion_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_completion_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_completion_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._3p_completion_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._3q.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3q.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3q.si, recovery.tMp0C0AQSb-jFrqFryYFnA._3r.cfe, recovery.tMp0C0AQSb-jFrqFryYFnA._3r.cfs, recovery.tMp0C0AQSb-jFrqFryYFnA._3r.si, recovery.tMp0C0AQSb-jFrqFryYFnA._v.dii, recovery.tMp0C0AQSb-jFrqFryYFnA._v.dim, recovery.tMp0C0AQSb-jFrqFryYFnA._v.fdx, recovery.tMp0C0AQSb-jFrqFryYFnA._v.fnm, recovery.tMp0C0AQSb-jFrqFryYFnA._v.nvd, recovery.tMp0C0AQSb-jFrqFryYFnA._v.nvm, recovery.tMp0C0AQSb-jFrqFryYFnA._v.si, recovery.tMp0C0AQSb-jFrqFryYFnA._v_8.liv, recovery.tMp0C0AQSb-jFrqFryYFnA._v_Lucene50_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._v_Lucene50_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._v_Lucene50_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._v_Lucene50_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._v_Lucene50_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA._v_Lucene70_0.dvd, recovery.tMp0C0AQSb-jFrqFryYFnA._v_Lucene70_0.dvm, recovery.tMp0C0AQSb-jFrqFryYFnA._v_completion_0.cmp, recovery.tMp0C0AQSb-jFrqFryYFnA._v_completion_0.doc, recovery.tMp0C0AQSb-jFrqFryYFnA._v_completion_0.lkp, recovery.tMp0C0AQSb-jFrqFryYFnA._v_completion_0.pay, recovery.tMp0C0AQSb-jFrqFryYFnA._v_completion_0.pos, recovery.tMp0C0AQSb-jFrqFryYFnA._v_completion_0.tim, recovery.tMp0C0AQSb-jFrqFryYFnA._v_completion_0.tip, recovery.tMp0C0AQSb-jFrqFryYFnA.segments_v, write.lock]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:670) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:627) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434) ~[lucene-core-7.3.1.jar:7.3.1 ae0705edb59eaa567fe13ed3a222fdadc7153680 - caomanhdat - 2018-05-09 09:27:24]
at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:122) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.store.Store.readSegmentsInfo(Store.java:207) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.store.Store.access$200(Store.java:134) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.store.Store$MetadataSnapshot.loadMetadata(Store.java:864) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.store.Store$MetadataSnapshot.<init>(Store.java:797) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.store.Store.getMetadata(Store.java:293) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.index.shard.IndexShard.snapshotStoreMetadata(IndexShard.java:1138) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.listStoreMetaData(TransportNodesListShardStoreMetaData.java:125) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:109) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.indices.store.TransportNodesListShardStoreMetaData.nodeOperation(TransportNodesListShardStoreMetaData.java:61) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction.nodeOperation(TransportNodesAction.java:140) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:260) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.action.support.nodes.TransportNodesAction$NodeTransportHandler.messageReceived(TransportNodesAction.java:256) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:246) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:304) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1592) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) ~[elasticsearch-6.3.0.jar:6.3.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.0.jar:6.3.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_162]
... 1 more
关于如何从该状态启动群集并重新运行的任何想法？
每次我尝试对集群进行更改时，都会收到超时，提示请求无法及时完成（读取查询工作正常）。 这似乎是Elasticsearcch中的一个严重错误，群集任务上应该有某种超时，以防止它们阻塞群集或至少是一种手动终止任务的方法。
（我们通过迁移到新的Elasticsearch集群解决了当前问题，但是如果在缩小集群规模时发生此错误，对我们来说将是一个大问题）