Question

我的目标是重新索引一个包含1000万个分片的索引，以便更改字段映射，以便进行重要的术语分析。

我的问题是我在使用NEST库执行重新索引时遇到问题，而且文档非常有限。如果可能，我需要使用以下示例：

http://nest.azurewebsites.net/nest/search/scroll.html

http://nest.azurewebsites.net/nest/core/bulk.html

Answer 1

NEST提供了一个很好的Reindex方法，尽管缺少文档。我使用这种特殊的WinForms代码以非常粗略和准备的方式使用它。

    private ElasticClient client;
    private double count;

    private void reindex_Completed()
    {
        MessageBox.Show("Done!");
    }

    private void reindex_Next(IReindexResponse<object> obj)
    {
        count += obj.BulkResponse.Items.Count();
        var progress = 100 * count / (double)obj.SearchResponse.Total;
        progressBar1.Value = (int)progress;
    }

    private void reindex_Error(Exception ex)
    {
        MessageBox.Show(ex.ToString());
    }

    private void button1_Click(object sender, EventArgs e)
    {
        count = 0;

        var reindex = client.Reindex<object>(r => r.FromIndex(fromIndex.Text).NewIndexName(toIndex.Text).Scroll("10s"));

        var o = new ReindexObserver<object>(onError: reindex_Error, onNext: reindex_Next, completed: reindex_Completed);
        reindex.Subscribe(o);
    }

我刚刚发现博客文章向我展示了如何做到这一点：http://thomasardal.com/elasticsearch-migrations-with-c-and-nest/

Answer 2

不幸的是NEST实现并不像我预期的那样。在我看来，它可能是最常见的用例有点过度设计。

很多人只想更新他们的映射，零停机时间......

在我的情况下 - 我已经开始使用其所有设置和映射来创建索引，但是NEST坚持在重建索引时必须创建一个新索引。这还有很多其他的事情。太多其他事情。

我发现直接实施起来要简单得多 - 因为NEST已经有Search，Scroll和Bulk方法。（这是从NEST的实施中采用的）：

// Assuming you have already created and setup the index yourself
public void Reindex(ElasticClient client, string aliasName, string currentIndexName, string nextIndexName)
{
    Console.WriteLine("Reindexing documents to new index...");
    var searchResult = client.Search<object>(s => s.Index(currentIndexName).AllTypes().From(0).Size(100).Query(q => q.MatchAll()).SearchType(SearchType.Scan).Scroll("2m"));
    if (searchResult.Total <= 0)
    {
        Console.WriteLine("Existing index has no documents, nothing to reindex.");
    }
    else
    {
        var page = 0;
        IBulkResponse bulkResponse = null;
        do
        {
            var result = searchResult;
            searchResult = client.Scroll<object>(s => s.Scroll("2m").ScrollId(result.ScrollId));
            if (searchResult.Documents != null && searchResult.Documents.Any())
            {
                searchResult.ThrowOnError("reindex scroll " + page);
                bulkResponse = client.Bulk(b =>
                {
                    foreach (var hit in searchResult.Hits)
                    {
                        b.Index<object>(bi => bi.Document(hit.Source).Type(hit.Type).Index(nextIndexName).Id(hit.Id));
                    }

                    return b;
                }).ThrowOnError("reindex page " + page);
                Console.WriteLine("Reindexing progress: " + (page + 1) * 100);
            }

            ++page;
        }
        while (searchResult.IsValid && bulkResponse != null && bulkResponse.IsValid && searchResult.Documents != null && searchResult.Documents.Any());
        Console.WriteLine("Reindexing complete!");
    }

    Console.WriteLine("Updating alias to point to new index...");
    client.Alias(a => a
        .Add(aa => aa.Alias(aliasName).Index(nextIndexName))
        .Remove(aa => aa.Alias(aliasName).Index(currentIndexName)));

    // TODO: Don't forget to delete the old index if you want
}

如果您需要ThrowOnError扩展方法：

public static T ThrowOnError<T>(this T response, string actionDescription = null) where T : IResponse
{
    if (!response.IsValid)
    {
        throw new CustomExceptionOfYourChoice(actionDescription == null ? string.Empty : "Failed to " + actionDescription + ": " + response.ServerError.Error);
    }

    return response;
}

Answer 3

我是第二个Ben Wilde的回答。最好完全控制索引创建和重新索引过程。

Ben代码中缺少的是对父母/子女关系的支持。这是我的代码来解决这个问题：

替换以下行：

foreach (var hit in searchResult.Hits)
{
    b.Index<object>(bi => bi.Document(hit.Source).Type(hit.Type).Index(nextIndexName).Id(hit.Id));
}

有了这个：

foreach (var hit in searchResult.Hits)
{
    var jo = hit.Source as JObject;
    JToken jt;
    if(jo != null && jo.TryGetValue("parentId", out jt))
    {
        // Document is child-document => add parent reference
        string parentId = (string)jt;
        b.Index<object>(bi => bi.Document(hit.Source).Type(hit.Type).Index(nextIndexName).Id(hit.Id).Parent(parentId));
    }
    else
    {
        b.Index<object>(bi => bi.Document(hit.Source).Type(hit.Type).Index(nextIndexName).Id(hit.Id));
    }                                
}

NEST Elasticsearch Reindex示例

3 个答案: