Question

首先，这是一个双节点集群，每个集群都有＆＃34; -Xms256m -Xmx1g -Xss256k＆＃34; （鉴于机器有8G＆＃34;这是非常糟糕的。

[2015-04-07 16:19:58,235][INFO ][monitor.jvm              ] [NODE1] [gc][ParNew][3246454][64605] duration [822ms], collections [1]/[4.3s], total [822ms]/[21m], memory [966.1mb]->[766.9mb]/[990.7mb], all_pools {[Code Cache] [13.1mb]->[13.1mb]/[48mb]}{[Par Eden Space] [266.2mb]->[75.6mb]/[266.2mb]}{[Par Survivor Space] [8.9mb]->[0b]/[33.2mb]}{[CMS Old Gen] [690.8mb]->[691.2mb]/[691.2mb]}{[CMS Perm Gen] [33.6mb]->[33.6mb]/[82mb]}
[2015-04-07 16:28:02,550][WARN ][transport.netty          ] [NODE1] exception caught on netty layer [[id: 0x03d14f1c, /10.0.6.100:36055 => /10.0.6.105:9300]]
java.lang.OutOfMemoryError: Java heap space
        at org.apache.lucene.util.PriorityQueue.initialize(PriorityQueue.java:108)
        at org.elasticsearch.search.controller.ScoreDocQueue.<init>(ScoreDocQueue.java:32)
....
[2015-04-07 21:55:54,743][WARN ][transport.netty          ] [NODE1] exception caught on netty layer [[id: 0xeea0018c, /10.0.6.100:36059 => /10.0.6.105:9300]]
java.lang.OutOfMemoryError: Java heap space
[2015-04-07 21:59:26,774][WARN ][transport.netty          ] [NODE1] exception caught on netty layer [[id: 0x576557fa, /10.0.6.100:36054 => /10.0.6.105:9300]]
...
[2015-04-07 22:51:05,890][WARN ][transport.netty          ] [NODE1] exception caught on netty layer [[id: 0x67f11ffe, /10.0.6.100:36052 => /10.0.
6.105:9300]]
org.elasticsearch.common.netty.handler.codec.frame.TooLongFrameException: transport content length received [1.5gb] exceeded [891.6mb]
[2015-04-07 22:51:05,973][WARN ][cluster.action.shard     ] [NODE1] sending failed shard for [test_index][15], nod
e[xvpLmlJkRSmZNj-pa_xUNA], [P], s[STARTED], reason [engine failure, message [OutOfMemoryError[Java heap space]]]

然后重新加入（我重新启动节点105）

[2015-04-07 22:59:11,095][INFO ][cluster.service          ] [NODE1] removed {[NODE2][GMBDo5K7RMGSgiIwZE7H8w][inet[/10.0.6.105:9300]],}, reason: zen-disco-node_failed([NODE7][GMBDo5K7RMGSgiIwZE7H8w][inet[/10.0.6.105:9300]]), reason transport disconnected (with verified connect)
[2015-04-07 22:59:30,954][INFO ][cluster.service          ] [NODE1] added {[NODE2][mMWcFGhVQY-aBR2r9DO3_A][inet[/10.0.6.105:9300]],}, reason: zen-disco-receive(join from node[[NODE7][mMWcFGhVQY-aBR2r9DO3_A][inet[/10.0.6.105:9300]]])
[2015-04-07 23:11:39,717][WARN ][transport.netty          ] [NODE1] exception caught on netty layer [[id: 0x14a605ce, /10.0.6.100:36201 => /10.0.6.105:9300]]
java.lang.OutOfMemoryError: Java heap space
[2015-04-07 23:16:04,963][WARN ][transport.netty          ] [NODE1] exception caught on netty layer [[id: 0x5a6d934d, /10.0.6.100:36196 => /10.0.6.105:9300]]
java.lang.OutOfMemoryError: Java heap space

所以我不知道如何解释＆＃34;＆gt;＆＃34;部分。谁真的失去了记忆？ NODE 1（10.0.6.100）？为什么端口9300？我的API最初与NODE1对话，所以在这种情况下它是否意味着NODE1向NODE2发送批量数据请求？这是第二天发生的事情

来自NODE1日志：

[2015-04-08 09:02:46,410][INFO ][cluster.service          ] [NODE1] removed {[NODE2][mMWcFGhVQY-aBR2r9DO3_A][inet[/10.0.6.105:9300]],}, reason: zen-disco-node_failed([NODE2][mMWcFGhVQY-aBR2r9DO3_A][inet[/10.0.6.105:9300]]), reason failed to ping, tried [3] times, each with maximum [30s] timeout
[2015-04-08 09:03:27,554][WARN ][search.action            ] [NODE1] Failed to send release search context
org.elasticsearch.transport.NodeDisconnectedException: [NODE2][inet[/10.0.6.105:9300]][search/freeContext] disconnected
....
Caused by: org.elasticsearch.transport.NodeNotConnectedException: [NODE2][inet[/10.0.6.105:9300]] Node not connected

但是在NODE2日志中，04-08只有几行，但是这样：

[2015-04-08 09:09:13,797][INFO ][discovery.zen            ] [NODE2] master_left [[NDOE1][xvpLmlJkRSmZNj-pa_xUNA][inet[/10.0.6.100:9300]]], reason [do not exists on master, act as master failure]

那究竟谁失败了？我在这里很困惑：|抱歉。任何帮助表示赞赏。我知道NODE1有一个非常长的GC（MarkSweep是3小时+直到昨晚我的双节点集群完全重启）。

Answer 1

日志的第一部分涉及Elasticsearch垃圾收集日志记录格式

[2015-04-07 16:19:58,235][INFO][monitor.jvm][NODE1]

垃圾收集运行
```
[gc] 
```
新的并行垃圾收集器
```
[ParNew]
```
GC耗时822毫秒
```
duration [822ms], 
```
一次集合运行，共计4.3秒
```
collections [1]/[4.3s]
```
池的使用次数＆＃39;内存＆＃39;以前是966.1mb，现在是766.9mb，总池大小为990.7mb
```
memory [966.1mb]->[766.9mb]/[990.7mb], 
```
池＆＃39;代码缓存＆＃39;
的使用数字
```
[Code Cache] [13.1mb]->[13.1mb]/[48mb]
```
游泳池使用数字＆＃39; Par Eden Space＆＃39;
```
[Par Eden Space] [266.2mb]->[75.6mb]/[266.2mb]
```
泳池使用数字＆＃39; Par Survivor Space＆＃39;
```
[Par Survivor Space] [8.9mb]->[0b]/[33.2mb]
```
游泳池使用数量＆＃39; CMS Old Gen＆＃39;
```
[CMS Old Gen] [690.8mb]->[691.2mb]/[691.2mb]
```
泳池使用数量＆＃39; CMS Perm Gen＆＃39;
```
[CMS Perm Gen] [33.6mb]->[33.6mb]/[82mb]
```

如果您注意到您的内存池几乎是1G。我希望这能给你一个提示！

解释ElasticSearch Out of Memory错误

1 个答案: