在ElasticSearch v1.1.0中插入新文档后,有时会出现noshardavailableactionexception错误。 使用
进行检查时curl 'localhost:9200/_cat/shards/cvk'
我得到了答案:cvk 0 p UNASSIGNED
使用命令重新启动弹性:
/etc/init.d/elasticsearch restart
一切正常。
ES在VPS ubuntu v12上运行,索引只有1个分片并且禁用了复制。
我发现"设备上没有空间"我的日志中的错误。但我的机器上有足够的光盘空间。我上传了一批1000份文件(每份文件大约512字节)。
如何解决冲洗问题?如果不可能,如何通过REST接口重新分配分片(不重启服务器)?
来自我的vps
Filesystem Size Used Avail Use% Mounted on
/dev/vda 20G 13G 6.6G 65% /
udev 237M 12K 237M 1% /dev
tmpfs 50M 216K 49M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 246M 0 246M 0% /run/shm
从vps显示错误
[2014-05-03 04:20:20,088][WARN ][index.translog ] [Molecule Man] [cvk][0] failed to flush shard on translog threshold
org.elasticsearch.index.engine.FlushFailedEngineException: [cvk][0] Flush failed
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:829)
at org.elasticsearch.index.shard.service.InternalIndexShard.flush(InternalIndexShard.java:589)
at org.elasticsearch.index.translog.TranslogService$TranslogBasedFlush$1.run(TranslogService.java:194)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes0(Native Method)
at java.io.RandomAccessFile.writeBytes(RandomAccessFile.java:520)
at java.io.RandomAccessFile.write(RandomAccessFile.java:550)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:452)
at org.apache.lucene.store.BufferedChecksumIndexOutput.flushBuffer(BufferedChecksumIndexOutput.java:71)
at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)
at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:102)
at org.apache.lucene.store.BufferedChecksumIndexOutput.flush(BufferedChecksumIndexOutput.java:86)
at org.apache.lucene.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:92)
at org.elasticsearch.index.store.Store$StoreIndexOutput.writeBytes(Store.java:634)
at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
at org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:65)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.writeBlock(BlockTreeTermsWriter.java:970)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:579)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter$FindBlocks.freeze(BlockTreeTermsWriter.java:555)
at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:214)
at org.apache.lucene.util.fst.Builder.add(Builder.java:394)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.finishTerm(BlockTreeTermsWriter.java:1047)
at org.apache.lucene.index.FreqProxTermsWriterPerField.flush(FreqProxTermsWriterPerField.java:548)
at org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
at org.apache.lucene.index.TermsHash.flush(TermsHash.java:116)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:53)
at org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:81)
at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:465)
at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:506)
at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:616)
at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2864)
at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3022)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2989)
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:812)
... 5 more
[2014-05-03 04:20:20,321][WARN ][index.merge.scheduler ] [Molecule Man] [cvk][0] failed to merge
java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes0(Native Method)
at java.io.RandomAccessFile.writeBytes(RandomAccessFile.java:520)
at java.io.RandomAccessFile.write(RandomAccessFile.java:550)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:452)
at org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput.flushBuffer(RateLimitedFSDirectory.java:102)
at org.apache.lucene.store.BufferedChecksumIndexOutput.flushBuffer(BufferedChecksumIndexOutput.java:71)
at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)
at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:102)
at org.apache.lucene.store.BufferedChecksumIndexOutput.flush(BufferedChecksumIndexOutput.java:86)
at org.apache.lucene.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:92)
at org.elasticsearch.index.store.Store$StoreIndexOutput.writeBytes(Store.java:634)
at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
at org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:65)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.writeBlock(BlockTreeTermsWriter.java:980)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:767)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter$FindBlocks.freeze(BlockTreeTermsWriter.java:555)
at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:214)
at org.apache.lucene.util.fst.Builder.add(Builder.java:394)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.finishTerm(BlockTreeTermsWriter.java:1047)
at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$WrappedTermsConsumer.finishTerm(BloomFilterPostingsFormat.java:439)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:112)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716)
at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122)
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89)
at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930)
at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47)
at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926)
at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2014-05-03 04:20:20,382][WARN ][index.engine.internal ] [Molecule Man] [cvk][0] failed engine
org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:92)
at org.elasticsearch.index.merge.EnableMergeScheduler.merge(EnableMergeScheduler.java:71)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1936)
at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:1930)
at org.elasticsearch.index.merge.Merges.maybeMerge(Merges.java:47)
at org.elasticsearch.index.engine.internal.InternalEngine.maybeMerge(InternalEngine.java:926)
at org.elasticsearch.index.shard.service.InternalIndexShard$EngineMerger$1.run(InternalIndexShard.java:966)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: No space left on device
at java.io.RandomAccessFile.writeBytes0(Native Method)
at java.io.RandomAccessFile.writeBytes(RandomAccessFile.java:520)
at java.io.RandomAccessFile.write(RandomAccessFile.java:550)
at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:452)
at org.apache.lucene.store.RateLimitedFSDirectory$RateLimitedIndexOutput.flushBuffer(RateLimitedFSDirectory.java:102)
at org.apache.lucene.store.BufferedChecksumIndexOutput.flushBuffer(BufferedChecksumIndexOutput.java:71)
at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:113)
at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:102)
at org.apache.lucene.store.BufferedChecksumIndexOutput.flush(BufferedChecksumIndexOutput.java:86)
at org.apache.lucene.store.BufferedIndexOutput.writeBytes(BufferedIndexOutput.java:92)
at org.elasticsearch.index.store.Store$StoreIndexOutput.writeBytes(Store.java:634)
at org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
at org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:65)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.writeBlock(BlockTreeTermsWriter.java:980)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.writeBlocks(BlockTreeTermsWriter.java:767)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter$FindBlocks.freeze(BlockTreeTermsWriter.java:555)
at org.apache.lucene.util.fst.Builder.freezeTail(Builder.java:214)
at org.apache.lucene.util.fst.Builder.add(Builder.java:394)
at org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter.finishTerm(BlockTreeTermsWriter.java:1047)
at org.elasticsearch.index.codec.postingsformat.BloomFilterPostingsFormat$WrappedTermsConsumer.finishTerm(BloomFilterPostingsFormat.java:439)
at org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:112)
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:383)
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:106)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4119)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3716)
at org.apache.lucene.index.TrackingSerialMergeScheduler.merge(TrackingSerialMergeScheduler.java:122)
at org.elasticsearch.index.merge.scheduler.SerialMergeSchedulerProvider$CustomSerialMergeScheduler.merge(SerialMergeSchedulerProvider.java:89)
... 9 more
[2014-05-03 04:20:20,490][DEBUG][action.bulk ] [Molecule Man] [cvk][0] failed to execute bulk item (index) index {[cvk][public][22017747], source[{"public":"22017747","name":"Private community | VK","desc":"\"\"","vol":0,"priv":null,"org":null,"phone":null,"email":null,"url":"5ghj6","wall":1,"post":null,"like":null,"share":null}]}
org.elasticsearch.index.engine.IndexFailedEngineException: [cvk][0] Index failed for [public#22017747]
at org.elasticsearch.index.engine.internal.InternalEngine.index(InternalEngine.java:483)
at org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:396)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:401)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:157)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:645)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:659)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1525)
at org.elasticsearch.index.engine.internal.InternalEngine.innerIndex(InternalEngine.java:532)
at org.elasticsearch.index.engine.internal.InternalEngine.index(InternalEngine.java:470)
... 8 more
[2014-05-03 04:20:20,493][DEBUG][action.bulk ] [Molecule Man] [cvk][0], node[Sk1Eoi84TDW9anq_zQsNJg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.bulk.BulkShardRequest@61204bff]
java.lang.NullPointerException
at org.elasticsearch.action.bulk.TransportShardBulkAction.applyVersion(TransportShardBulkAction.java:617)
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:178)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
[2014-05-03 04:20:20,534][WARN ][cluster.action.shard ] [Molecule Man] [cvk][0] sending failed shard for [cvk][0], node[Sk1Eoi84TDW9anq_zQsNJg], [P], s[STARTED], indexUUID [m0nqEEqXQu-rHc5ipn4ZPA], reason [engine failure, message [MergeException[java.io.IOException: No space left on device]; nested: IOException[No space left on device]; ]]
[2014-05-03 04:20:20,534][WARN ][cluster.action.shard ] [Molecule Man] [cvk][0] received shard failed for [cvk][0], node[Sk1Eoi84TDW9anq_zQsNJg], [P], s[STARTED], indexUUID [m0nqEEqXQu-rHc5ipn4ZPA], reason [engine failure, message [MergeException[java.io.IOException: No space left on device]; nested: IOException[No space left on device]; ]]
节点信息
indices: {
docs: {
count: 4439439
deleted: 0
}
store: {
size_in_bytes: 643890465
throttle_time_in_millis: 0
}
indexing: {
index_total: 2214686
index_time_in_millis: 1679906
index_current: 1
delete_total: 0
delete_time_in_millis: 0
delete_current: 0
}
get: {
total: 0
time_in_millis: 0
exists_total: 0
exists_time_in_millis: 0
missing_total: 0
missing_time_in_millis: 0
current: 0
}
search: {
open_contexts: 0
query_total: 0
query_time_in_millis: 0
query_current: 0
fetch_total: 0
fetch_time_in_millis: 0
fetch_current: 0
}
merges: {
current: 0
current_docs: 0
current_size_in_bytes: 0
total: 23
total_time_in_millis: 1081333
total_docs: 15716810
total_size_in_bytes: 5938832547
}
refresh: {
total: 8
total_time_in_millis: 0
}
flush: {
total: 202
total_time_in_millis: 677609
}
warmer: {
current: 0
total: 2
total_time_in_millis: 15
}
filter_cache: {
memory_size_in_bytes: 0
evictions: 0
}
id_cache: {
memory_size_in_bytes: 0
}
fielddata: {
memory_size_in_bytes: 0
evictions: 0
}
percolate: {
total: 0
time_in_millis: 0
current: 0
memory_size_in_bytes: -1
memory_size: -1b
queries: 0
}
completion: {
size_in_bytes: 0
}
segments: {
count: 18
memory_in_bytes: 38866707
}
translog: {
operations: 0
size_in_bytes: 0
}
}
os: {
timestamp: 1399114654034
uptime_in_millis: 701756
load_average: [
0
0.01
0.05
]
cpu: {
sys: 0
user: 0
idle: 99
usage: 0
stolen: 0
}
mem: {
free_in_bytes: 34357248
used_in_bytes: 480374784
free_percent: 33
used_percent: 66
actual_free_in_bytes: 172974080
actual_used_in_bytes: 341757952
}
swap: {
used_in_bytes: 0
free_in_bytes: 0
}
}
process: {
timestamp: 1399114654035
open_file_descriptors: 103
cpu: {
percent: 0
sys_in_millis: 118480
user_in_millis: 2057680
total_in_millis: 2176160
}
mem: {
resident_in_bytes: 263897088
share_in_bytes: 6635520
total_virtual_in_bytes: 1609924608
}
}
jvm: {
timestamp: 1399114654035
uptime_in_millis: 43582377
mem: {
heap_used_in_bytes: 80238424
heap_used_percent: 52
heap_committed_in_bytes: 152043520
heap_max_in_bytes: 152043520
non_heap_used_in_bytes: 42873536
non_heap_committed_in_bytes: 66764800
pools: {
young: {
used_in_bytes: 15877936
max_in_bytes: 41943040
peak_used_in_bytes: 41943040
peak_max_in_bytes: 41943040
}
survivor: {
used_in_bytes: 1463048
max_in_bytes: 5242880
peak_used_in_bytes: 5242880
peak_max_in_bytes: 5242880
}
old: {
used_in_bytes: 62897440
max_in_bytes: 104857600
peak_used_in_bytes: 104857600
peak_max_in_bytes: 104857600
}
}
}
threads: {
count: 36
peak_count: 40
}
gc: {
collectors: {
young: {
collection_count: 7359
collection_time_in_millis: 116960
}
old: {
collection_count: 2693
collection_time_in_millis: 131864
}
}
}
buffer_pools: {
direct: {
count: 16
used_in_bytes: 2694367
total_capacity_in_bytes: 2694367
}
mapped: {
count: 83
used_in_bytes: 635281868
total_capacity_in_bytes: 635281868
}
}
}
thread_pool: {
generic: {
threads: 2
queue: 0
active: 0
rejected: 0
largest: 6
completed: 9045
}
index: {
threads: 0
queue: 0
active: 0
rejected: 0
largest: 0
completed: 0
}
get: {
threads: 0
queue: 0
active: 0
rejected: 0
largest: 0
completed: 0
}
snapshot: {
threads: 1
queue: 0
active: 0
rejected: 0
largest: 1
completed: 442
}
merge: {
threads: 1
queue: 0
active: 0
rejected: 0
largest: 1
completed: 1297
}
suggest: {
threads: 0
queue: 0
active: 0
rejected: 0
largest: 0
completed: 0
}
bulk: {
threads: 1
queue: 0
active: 0
rejected: 0
largest: 1
completed: 2213
}
optimize: {
threads: 0
queue: 0
active: 0
rejected: 0
largest: 0
completed: 0
}
warmer: {
threads: 1
queue: 0
active: 0
rejected: 0
largest: 1
completed: 262
}
flush: {
threads: 1
queue: 0
active: 0
rejected: 0
largest: 1
completed: 205
}
search: {
threads: 0
queue: 0
active: 0
rejected: 0
largest: 0
completed: 0
}
percolate: {
threads: 0
queue: 0
active: 0
rejected: 0
largest: 0
completed: 0
}
management: {
threads: 5
queue: 0
active: 1
rejected: 0
largest: 5
completed: 7511
}
refresh: {
threads: 0
queue: 0
active: 0
rejected: 0
largest: 0
completed: 0
}
}
network: {
tcp: {
active_opens: 366578
passive_opens: 32901
curr_estab: 34
in_segs: 450996588
out_segs: 379209662
retrans_segs: 35059
estab_resets: 2230
attempt_fails: 1298
in_errs: 62
out_rsts: 6939
}
}
fs: {
timestamp: 1399114654035
total: {
total_in_bytes: 21003628544
free_in_bytes: 8092241920
available_in_bytes: 7018500096
disk_reads: 4208794
disk_writes: 5227010
disk_io_op: 9435804
disk_read_size_in_bytes: 285034193920
disk_write_size_in_bytes: 518983745536
disk_io_size_in_bytes: 804017939456
}
data: [
{
path: /var/lib/elasticsearch/elasticsearch/nodes/0
mount: /
dev: /dev/vda
total_in_bytes: 21003628544
free_in_bytes: 8092241920
available_in_bytes: 7018500096
disk_reads: 4208794
disk_writes: 5227010
disk_io_op: 9435804
disk_read_size_in_bytes: 285034193920
disk_write_size_in_bytes: 518983745536
disk_io_size_in_bytes: 804017939456
}
]
}
transport: {
server_open: 13
rx_count: 0
rx_size_in_bytes: 0
tx_count: 0
tx_size_in_bytes: 0
}
http: {
current_open: 6
total_opened: 2431
}
fielddata_breaker: {
maximum_size_in_bytes: 121634816
maximum_size: 116mb
estimated_size_in_bytes: 0
estimated_size: 0b
overhead: 1.03
}
现在,即使ES重新启动,分片仍然不可用。这是日志
[2014-05-03 07:10:18,903][INFO ][gateway ] [Mys-Tech] recovered [2] indices into cluster_state
[2014-05-03 07:10:18,905][INFO ][node ] [Mys-Tech] started
[2014-05-03 07:10:41,334][WARN ][indices.cluster ] [Mys-Tech] [cvk][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [cvk][0] failed recovery
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:256)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.FlushNotAllowedEngineException: [cvk][0] already flushing...
at org.elasticsearch.index.engine.internal.InternalEngine.flush(InternalEngine.java:745)
at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryFinalization(InternalIndexShard.java:716)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:250)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:197)
... 3 more
[2014-05-03 07:10:44,601][WARN ][cluster.action.shard ] [Mys-Tech] [cvk][0] sending failed shard for [cvk][0], node[gknU3JzTRviIpDi4O-rc6A], [P], s[INITIALIZING], indexUUID [m0nqEEqXQu-rHc5ipn4ZPA], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[cvk][0] failed recovery]; nested: FlushNotAllowedEngineException[[cvk][0] already flushing...]; ]]
[2014-05-03 07:10:44,602][WARN ][cluster.action.shard ] [Mys-Tech] [cvk][0] received shard failed for [cvk][0], node[gknU3JzTRviIpDi4O-rc6A], [P], s[INITIALIZING], indexUUID [m0nqEEqXQu-rHc5ipn4ZPA], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[cvk][0] failed recovery]; nested: FlushNotAllowedEngineException[[cvk][0] already flushing...]; ]]
答案 0 :(得分:2)
首先,您似乎正在运行Elasticsearch,并在根分区上创建索引:
Filesystem Size Used Avail Use% Mounted on
/dev/vda 20G 13G 6.6G 65% /
udev 237M 12K 237M 1% /dev
tmpfs 50M 216K 49M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 246M 0 246M 0% /run/shm
如果你有能力安装另一个驱动器,通常不是最好的主意。
您在Lucene索引段合并期间失败,这通常需要大量可用空间。由于只有20G的非常小的分区,磁盘空间使用率已达到65%,因此您可以轻松地耗尽空间,特别是因为您同时竞争所有其他进程的磁盘需求。有关管理和配置Elasticsearch合并策略的更多详细信息:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-merge.html
您可能无法在20GB分区(也是根分区)上可靠地索引和管理9 GB的数据,尤其是在您大量更改数据时。您可以尝试将其设置为避免/减少可能有助于磁盘空间的段合并,但这仍然可能无效。
关于它为什么占用尽可能多的空间,这取决于您如何映射数据,但一般来说Elasticsearch默认存储所有数据的副本,其中包含所有数据的原始形式,以及所有数据每个字段的索引数。
如果您确实需要适应20GB系统,我会仔细查看您的映射,看看哪些字段不能索引或不存储 -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-source-field.html http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-all-field.html
答案 1 :(得分:0)
问题确实存在于光盘领域。由于未知原因,ES占用所有可用磁盘空间。 这里发生了什么:
我通过索引中的批量API添加了大约75000个文档(全部成功)。
然后根本不要触摸ES。并监控磁盘空间。
在5分钟内,所有空间都被/ var / lib / elasticsearch / elasticsearch / nodes / 0 / indeces / cvk / 0 / index /中的少量文件占用 最大的空间文件_3ya.fdt(3gig) 在丢失碎片之前,有一些名为_3ya_es090_0的文件,扩展名为.tim .pos .doc,每个大约400mb。 丢失碎片之后,所有这些文件都消失了。
所以显而易见的解决方案是添加光盘内存。
但是新问题:
为什么ES需要x10磁盘空间而不是正在添加的数据大小?
有没有办法知道何时停止在现有分片中添加新文档?
如果我们创建多个分片而不是一个分片会有帮助吗?
如何从当前服务器获取最大值的任何其他建议?服务器有20GB的空间。我们只需要为小型研究索引大约9gig的数据。