我们最近开始在我们的应用程序和应用程序之一上使用elasticsearch(V-1.5)。开始面临以下问题..
生成日志(异常非常高,即5分钟5GB日志)
高CPU利用率
尽管删除了日志,但未释放磁盘空间。需要努力 启动服务器以释放日志空间。
系统配置
不知道出了什么问题。任何线索......
Caused by: org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [.marvel-2015.03.16][0] shard allocated for local recovery (post ap
i), should exist, but doesn't, current files: []
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:131)
... 4 more
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(least_used[rate_limited(default(mmapfs(/DATA/www/elast
icsearch/data/elasticsearch/nodes/0/indices/.marvel-2015.03.16/0/index),niofs(/DATA/www/elasticsearch/data/elasticsearch/nodes/0/indices/.marvel-2
015.03.16/0/index)), type=MERGE, rate=20.0)]): files: []
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:881)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:769)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:458)
at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:89)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:122)
... 4 more
[2015-04-04 15:18:01,064][WARN ][cluster.action.shard ] [Trick Shot] [.marvel-2015.03.16][0] sending failed shard for [.marvel-2015.03.16][0],
node[4YS3jneRTn624zDl6C29Zg], [P], s[INITIALIZING], indexUUID [FKpoioArTI-KxevMFlQKAQ], reason [Failed to start shard, message [IndexShardGateway
RecoveryException[[.marvel-2015.03.16][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[.marve
l-2015.03.16][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no s
egments* file found in store(least_used[rate_limited(default(mmapfs(/DATA/www/elasticsearch/data/elasticsearch/nodes/0/indices/.marvel-2015.03.16/
0/index),niofs(/DATA/www/elasticsearch/data/elasticsearch/nodes/0/indices/.marvel-2015.03.16/0/index)), type=MERGE, rate=20.0)]): files: []]; ]]
[2015-04-04 15:18:01,064][WARN ][cluster.action.shard ] [Trick Shot] [.marvel-2015.03.16][0] received shard failed for [.marvel-2015.03.16][0]
, node[4YS3jneRTn624zDl6C29Zg], [P], s[INITIALIZING], indexUUID [FKpoioArTI-KxevMFlQKAQ], reason [Failed to start shard, message [IndexShardGatewa
yRecoveryException[[.marvel-2015.03.16][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[.marv
el-2015.03.16][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no
segments* file found in store(least_used[rate_limited(default(mmapfs(/DATA/www/elasticsearch/data/elasticsearch/nodes/0/indices/.marvel-2015.03.16
/0/index),niofs(/DATA/www/elasticsearch/data/elasticsearch/nodes/0/indices/.marvel-2015.03.16/0/index)), type=MERGE, rate=20.0)]): files: []]; ]]
[2015-04-04 15:18:01,079][WARN ][cluster.action.shard ] [Trick Shot] [.marvel-2015.03.16][0] received shard failed for [.marvel-2015.03.16][0]
, node[4YS3jneRTn624zDl6C29Zg], [P], s[INITIALIZING], indexUUID [FKpoioArTI-KxevMFlQKAQ], reason [master [Trick Shot][4YS3jneRTn624zDl6C29Zg][E2pr
o-ESP-06][inet[/10.9.46.65:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-04-04 15:18:01,080][WARN ][indices.cluster ] [Trick Shot] [.marvel-2015.03.15][0] sending failed shard after recovery failure
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [.marvel-2015.03.15][0] failed to fetch index version after copying it over
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:157)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [.marvel-2015.03.15][0] shard allocated for local recovery (post ap
i), should exist, but doesn't, current files: []
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:131)
... 4 more
Caused by: org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(least_used[rate_limited(default(mmapfs(/DATA/www/elast
icsearch/data/elasticsearch/nodes/0/indices/.marvel-2015.03.15/0/index),niofs(/DATA/www/elasticsearch/data/elasticsearch/nodes/0/indices/.marvel-2
015.03.15/0/index)), type=MERGE, rate=20.0)]): files: []
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:881)
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:769)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:458)
at org.elasticsearch.common.lucene.Lucene.readSegmentInfos(Lucene.java:89)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:122)
... 4 more
[2015-04-04 15:18:01,080][WARN ][cluster.action.shard ] [Trick Shot] [.marvel-2015.03.15][0] sending failed shard for [.marvel-2015.03.15][0],
node[4YS3jneRTn624zDl6C29Zg], [P], s[INITIALIZING], indexUUID [uFZ3KaB6Te2igqubyvkFHA], reason [Failed to start shard, message [IndexShardGateway
RecoveryException[[.marvel-2015.03.15][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[.marve
l-2015.03.15][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no s
egments* file found in store(least_used[rate_limited(default(mmapfs(/DATA/www/elasticsearch/data/elasticsearch/nodes/0/indices/.marvel-2015.03.15/
0/index),niofs(/DATA/www/elasticsearch/data/elasticsearch/nodes/0/indices/.marvel-2015.03.15/0/index)), type=MERGE, rate=20.0)]): files: []]; ]]
[2015-04-04 15:18:01,080][WARN ][cluster.action.shard ] [Trick Shot] [.marvel-2015.03.15][0] received shard failed for [.marvel-2015.03.15][0]
, node[4YS3jneRTn624zDl6C29Zg], [P], s[INITIALIZING], indexUUID [uFZ3KaB6Te2igqubyvkFHA], reason [Failed to start shard, message [IndexShardGatewa
yRecoveryException[[.marvel-2015.03.15][0] failed to fetch index version after copying it over]; nested: IndexShardGatewayRecoveryException[[.marv
el-2015.03.15][0] shard allocated for local recovery (post api), should exist, but doesn't, current files: []]; nested: IndexNotFoundException[no
segments* file found in store(least_used[rate_limited(default(mmapfs(/DATA/www/elasticsearch/data/elasticsearch/nodes/0/indices/.marvel-2015.03.15
/0/index),niofs(/DATA/www/elasticsearch/data/elasticsearch/nodes/0/indices/.marvel-2015.03.15/0/index)), type=MERGE, rate=20.0)]): files: []]; ]]
[2015-04-04 15:18:01,094][WARN ][cluster.action.shard ] [Trick Shot] [.marvel-2015.03.15][0] received shard failed for [.marvel-2015.03.15][0]
, node[4YS3jneRTn624zDl6C29Zg], [P], s[INITIALIZING], indexUUID [uFZ3KaB6Te2igqubyvkFHA], reason [master [Trick Shot][4YS3jneRTn624zDl6C29Zg][E2pr
o-ESP-06][inet[/10.9.46.65:9300]] marked shard as initializing, but shard is marked as failed, resend shard failure]
[2015-04-04 15:18:01,096][WARN ][indices.cluster ] [Trick Shot] [.marvel-2015.03.13][0] sending failed shard after recovery failure
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [.marvel-2015.03.13][0] failed to fetch index version after copying it over
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:157)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:112)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)