重新启动2节点elasticsearch集群，停机时间为零

时间：2014-12-07 17:28:33

标签： elasticsearch

我们一直在运行一个双节点ES群集（现在在 1.4.1 上），包含所有默认值和以下覆盖：

config.cluster.name = "..."
config.discovery.zen.ping_timeout = "5s";
config.discovery.zen.ping.multicast.enabled = false;
config.discovery.zen.ping.unicast.hosts = ["IP1", "IP2"];

最近，我们开始注意到，当我们通过http://127.0.0.1:9200/_cluster/nodes/_local/_shutdown请求关闭每个节点时，群集无法响应 30秒。

当主节点明确关闭时，另一个节点似乎不会立即恢复主节点的角色......而是继续尝试直到超过30秒（默认discovery.zen.fd.ping_timeout）超时。< / p>

在此期间，群集中有一个“没有主人”＆＃39;阻止并返回503到根节点请求：

{
"status" : 503,
"name" : "...",
"cluster_name" : "...",
"version" : {
"number" : "1.4.1",
"build_hash" : "89d3241d670db65f994242c8e8383b169779e2d4",
"build_timestamp" : "2014-11-26T15:49:29Z",
"build_snapshot" : false,
"lucene_version" : "4.10.2"
},
"tagline" : "You Know, for Search"
}

块级别为[＆＃34;写入＆＃34;，＆＃34;元数据＆＃34;]。

你可以在日志中看到这个：

[2014-12-04 17:46:16,000][INFO ][discovery.zen            ] [NODE_1] master_left [[NODE_0][VXtqWIw2Q2C9b5UHvWlZyQ][RD000D3A1024B8][inet[/100.72.14.37:9300]]], reason [shut_down]
[2014-12-04 17:46:16,012][WARN ][discovery.zen            ] [NODE_1] master left (reason = shut_down), current nodes: {[NODE_1][WoVynRBhQvSwvxNp1nj8kw][RD000D3A109006][inet[/100.78.140.38:9300]],}
[2014-12-04 17:46:16,012][INFO ][cluster.service          ] [NODE_1] removed {[NODE_0][VXtqWIw2Q2C9b5UHvWlZyQ][RD000D3A1024B8][inet[/100.72.14.37:9300]],}, reason: zen-disco-master_failed ([NODE_0][VXtqWIw2Q2C9b5UHvWlZyQ][RD000D3A1024B8][inet[/100.72.14.37:9300]])
[2014-12-04 17:46:16,497][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:18,358][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:19,508][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:20,384][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:21,150][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:21,915][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:22,540][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:23,384][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:23,900][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:24,572][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:26,794][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:27,783][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:28,441][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:29,330][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:30,393][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:31,264][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:31,905][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:32,608][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:35,572][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:36,529][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:37,295][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:37,911][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:38,661][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:39,411][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:40,032][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:40,643][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:41,505][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:41,927][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:42,630][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:43,380][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:44,193][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:44,963][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:45,824][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:46,511][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:46,574][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:47,278][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:48,028][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:48,373][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:48,811][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:49,530][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:49,530][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:50,155][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:50,405][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:51,030][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:51,170][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:51,889][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:51,938][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:52,530][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:52,561][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:53,406][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:53,596][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:53,908][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:54,353][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:54,587][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:55,056][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:55,712][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:56,322][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:56,806][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:56,962][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:57,791][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:57,806][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:58,228][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:58,447][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:59,088][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:46:59,355][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:46:59,775][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:47:00,400][DEBUG][action.admin.cluster.state] [NODE_1] observer: timeout notification from cluster service. timeout setting [30s], time since start [30s]
[2014-12-04 17:47:00,400][DEBUG][action.admin.cluster.state] [NODE_1] no known master node, scheduling a retry
[2014-12-04 17:47:00,619][INFO ][cluster.service          ] [NODE_1] new_master [NODE_1][WoVynRBhQvSwvxNp1nj8kw][RD000D3A109006][inet[/100.78.140.38:9300]], reason: zen-disco-join (elected_as_master)

我们如何强制当前节点在shutdown命令期间放弃其作为master的角色，以便其他节点可以立即恢复此责任并防止30秒无主块中断？我们已经尝试过各种瞬态＆＃39;群集更新呼叫迫使立即选举无济于事。

1 个答案:

答案 0 :(得分：3)

您需要在设置文件中将discovery.zen.rejoin_on_master_gone设置为false 默认情况下，此值为true，这意味着当主服务器离开而不是选择主服务器时，节点将重新启动集群连接过程。如果您将其设置为false，则一旦注意到旧主人已经离开（假设您的discovery.zen.minimum_master_nodes设置不是2），该节点就会选择自己为主人。

编辑添加警告：此设置已在Elasticsearch 1.4中添加，作为其修复程序的一部分，以帮助防止分裂大脑。如果您的群集容易出现这些（或者再次看到它们），那么您希望按原样保留此设置。

相关问题
最新问题