弹性搜索2.3.4停止分配没有明显原因的分片

时间:2016-07-26 08:54:56

标签: elasticsearch

我正在尝试将我们的弹性搜索群集从1.6升级到2.3.4。升级似乎有效,我可以看到分片开始在Kopf中发生 - 但在某些时候,分片分配似乎停止,许多分片未分配,并且日志中没有报告错误。通常我会留下1200/3800个未分配的分片。

我们有一个典型的3节点集群,我正在我的本地计算机上试用这个独立节点,所有3个节点都在我的本地计算机上运行。

我看到过类似的症状报告 - 请参阅https://t37.net/how-to-fix-your-elasticsearch-cluster-stuck-in-initializing-shards-mode.html 。这里的解决方案似乎是手动分配我已尝试(并且有效)的分片,但我不知道在这里解释弹性搜索的行为。我不想沿着这条路走下去,因为我希望我的集群在没有干预的情况下自动旋转。

还有https://github.com/elastic/elasticsearch/pull/14494似乎是用最新的ES版本解决的,所以不应该成为一个问题。

日志文件中没有错误 - 我已经将根级别记录到' DEBUG'为了看到我能做什么。对于每个未分配的分片,我可以看到如下所示的行(来自主节点日志):

[2016-07-26 09:18:04,859][DEBUG][gateway                  ] [germany] [index][4] found 0 allocations of [index][4], node[null], [P], v[0], s[UNASSIGNED], unassigned_info[[reason=CLUSTER_RECOVERED], at[2016-07-26T08:05:04.447Z]], highest version: [-1]
[2016-07-26 09:18:04,859][DEBUG][gateway                  ] [germany] [index][4]: not allocating, number_of_allocated_shards_found [0]

配置文件(删除了评论):

cluster.name: elasticsearch-jm-2.3.4
node.name: germany

script.inline: true
script.indexed: true

如果我在重新分配停止后查询群集运行状况 - 我得到以下响应:

http://localhost:9200/_cluster/health?pretty

cluster_name                     : elasticsearch-jm-2.3.4
status                           : red
timed_out                        : False
number_of_nodes                  : 3
number_of_data_nodes             : 3
active_primary_shards            : 1289
active_shards                    : 2578
relocating_shards                : 0
initializing_shards              : 0
unassigned_shards                : 1264
delayed_unassigned_shards        : 0
number_of_pending_tasks          : 0
number_of_in_flight_fetch        : 0
task_max_waiting_in_queue_millis : 0
active_shards_percent_as_number  : 67.10046850598647

进一步查询分片 - 使用未分配的分片过滤到一个索引。可以看出 - shard 0和4未分配,而shard 1 2和3已分配:

http://localhost:9200/_cat/shards

cs-payment-warn-2016.07.20      3 p STARTED       106  92.4kb 127.0.0.1 germany     
cs-payment-warn-2016.07.20      3 r STARTED       106  92.4kb 127.0.0.1 switzerland 
cs-payment-warn-2016.07.20      4 p UNASSIGNED                                      
cs-payment-warn-2016.07.20      4 r UNASSIGNED                                      
cs-payment-warn-2016.07.20      2 r STARTED       120  74.5kb 127.0.0.1 cyprus      
cs-payment-warn-2016.07.20      2 p STARTED       120  74.5kb 127.0.0.1 germany     
cs-payment-warn-2016.07.20      1 r STARTED       120  73.8kb 127.0.0.1 cyprus      
cs-payment-warn-2016.07.20      1 p STARTED       120  73.8kb 127.0.0.1 germany     
cs-payment-warn-2016.07.20      0 p UNASSIGNED                                      
cs-payment-warn-2016.07.20      0 r UNASSIGNED                                      

手动重新路由未分配的分片似乎有效 - (去掉结果集)

http://localhost:9200/_cluster/reroute 
POST:
{ 
    "dry_run": true, 
    "commands": [ 
        { 
            "allocate": { 
                "index": "cs-payment-warn-2016.07.20", 
                "shard": 4, 
                "node": "switzerland" , 
                "allow_primary": true
             } 
        } 
    ] 
}

Response:
{
    "acknowledged" : true,
    "state" : {
        "version" : 722,
        "state_uuid" : "Vw2vPoCMQk2ZosjzviD4TQ",
        "master_node" : "yhL7XXy-SKu_WAM-C33dzA",
        "blocks" : {},
        "nodes" : {},
        "routing_table" : {
            "indices" : {
                "cs-payment-warn-2016.07.20" : {
                    "shards" : {
                        "3" : [{
                                "state" : "STARTED",
                                "primary" : true,
                                "node" : "yhL7XXy-SKu_WAM-C33dzA",
                                "relocating_node" : null,
                                "shard" : 3,
                                "index" : "cs-payment-warn-2016.07.20",
                                "version" : 22,
                                "allocation_id" : {
                                    "id" : "x_Iq88hmTqiasrjW09hVuw"
                                }
                            }, {
                                "state" : "STARTED",
                                "primary" : false,
                                "node" : "1a8dgBscTUS3c7Pv4mN9CQ",
                                "relocating_node" : null,
                                "shard" : 3,
                                "index" : "cs-payment-warn-2016.07.20",
                                "version" : 22,
                                "allocation_id" : {
                                    "id" : "DF-EUEy_SpeUElnZI6cgsQ"
                                }
                            }
                        ],
                        "4" : [{
                                "state" : "INITIALIZING",
                                "primary" : true,
                                "node" : "1a8dgBscTUS3c7Pv4mN9CQ",
                                "relocating_node" : null,
                                "shard" : 4,
                                "index" : "cs-payment-warn-2016.07.20",
                                "version" : 1,
                                "allocation_id" : {
                                    "id" : "1tw7C7YPQsWwm_O-8mYHRg"
                                },
                                "unassigned_info" : {
                                    "reason" : "INDEX_CREATED",
                                    "at" : "2016-07-26T14:20:15.395Z",
                                    "details" : "force allocation from previous reason CLUSTER_RECOVERED, null"
                                }
                            }, {
                                "state" : "UNASSIGNED",
                                "primary" : false,
                                "node" : null,
                                "relocating_node" : null,
                                "shard" : 4,
                                "index" : "cs-payment-warn-2016.07.20",
                                "version" : 1,
                                "unassigned_info" : {
                                    "reason" : "CLUSTER_RECOVERED",
                                    "at" : "2016-07-26T11:24:11.868Z"
                                }
                            }
                        ],
                        "2" : [{
                                "state" : "STARTED",
                                "primary" : false,
                                "node" : "rlRQ2u0XQRqxWld-wSrOug",
                                "relocating_node" : null,
                                "shard" : 2,
                                "index" : "cs-payment-warn-2016.07.20",
                                "version" : 22,
                                "allocation_id" : {
                                    "id" : "eQ-_vWNbRp27So0iGSitmA"
                                }
                            }, {
                                "state" : "STARTED",
                                "primary" : true,
                                "node" : "yhL7XXy-SKu_WAM-C33dzA",
                                "relocating_node" : null,
                                "shard" : 2,
                                "index" : "cs-payment-warn-2016.07.20",
                                "version" : 22,
                                "allocation_id" : {
                                    "id" : "O1PU1_NVS8-uB2yBrG76MA"
                                }
                            }
                        ],
                        "1" : [{
                                "state" : "STARTED",
                                "primary" : false,
                                "node" : "rlRQ2u0XQRqxWld-wSrOug",
                                "relocating_node" : null,
                                "shard" : 1,
                                "index" : "cs-payment-warn-2016.07.20",
                                "version" : 24,
                                "allocation_id" : {
                                    "id" : "ZmxtOvorRVmndR15OJMkMA"
                                }
                            }, {
                                "state" : "STARTED",
                                "primary" : true,
                                "node" : "yhL7XXy-SKu_WAM-C33dzA",
                                "relocating_node" : null,
                                "shard" : 1,
                                "index" : "cs-payment-warn-2016.07.20",
                                "version" : 24,
                                "allocation_id" : {
                                    "id" : "ZNgzePThQxS-iqhRSXzZCw"
                                }
                            }
                        ],
                        "0" : [{
                                "state" : "UNASSIGNED",
                                "primary" : true,
                                "node" : null,
                                "relocating_node" : null,
                                "shard" : 0,
                                "index" : "cs-payment-warn-2016.07.20",
                                "version" : 0,
                                "unassigned_info" : {
                                    "reason" : "CLUSTER_RECOVERED",
                                    "at" : "2016-07-26T11:24:11.868Z"
                                }
                            }, {
                                "state" : "UNASSIGNED",
                                "primary" : false,
                                "node" : null,
                                "relocating_node" : null,
                                "shard" : 0,
                                "index" : "cs-payment-warn-2016.07.20",
                                "version" : 0,
                                "unassigned_info" : {
                                    "reason" : "CLUSTER_RECOVERED",
                                    "at" : "2016-07-26T11:24:11.868Z"
                                }
                            }
                        ]
                    }
                }

            },
            "routing_nodes" : {
                "unassigned" : [{
                        "state" : "UNASSIGNED",
                        "primary" : false,
                        "node" : null,
                        "relocating_node" : null,
                        "shard" : 4,
                        "index" : "cs-payment-warn-2016.07.20",
                        "version" : 1,
                        "unassigned_info" : {
                            "reason" : "CLUSTER_RECOVERED",
                            "at" : "2016-07-26T11:24:11.868Z"
                        }
                    }, {
                        "state" : "UNASSIGNED",
                        "primary" : true,
                        "node" : null,
                        "relocating_node" : null,
                        "shard" : 0,
                        "index" : "cs-payment-warn-2016.07.20",
                        "version" : 0,
                        "unassigned_info" : {
                            "reason" : "CLUSTER_RECOVERED",
                            "at" : "2016-07-26T11:24:11.868Z"
                        }
                    }, {
                        "state" : "UNASSIGNED",
                        "primary" : false,
                        "node" : null,
                        "relocating_node" : null,
                        "shard" : 0,
                        "index" : "cs-payment-warn-2016.07.20",
                        "version" : 0,
                        "unassigned_info" : {
                            "reason" : "CLUSTER_RECOVERED",
                            "at" : "2016-07-26T11:24:11.868Z"
                        }
                    }

                ]
            },
            "nodes" : {
                "rlRQ2u0XQRqxWld-wSrOug" : [{
                        "state" : "STARTED",
                        "primary" : false,
                        "node" : "rlRQ2u0XQRqxWld-wSrOug",
                        "relocating_node" : null,
                        "shard" : 2,
                        "index" : "cs-payment-warn-2016.07.20",
                        "version" : 22,
                        "allocation_id" : {
                            "id" : "eQ-_vWNbRp27So0iGSitmA"
                        }
                    }, {
                        "state" : "STARTED",
                        "primary" : false,
                        "node" : "rlRQ2u0XQRqxWld-wSrOug",
                        "relocating_node" : null,
                        "shard" : 1,
                        "index" : "cs-payment-warn-2016.07.20",
                        "version" : 24,
                        "allocation_id" : {
                            "id" : "ZmxtOvorRVmndR15OJMkMA"
                        }
                    }
                ]
            }

        }
    }
}

0 个答案:

没有答案