确保Scroll API的结果顺序

时间:2017-02-14 15:57:29

标签: elasticsearch

我正在尝试使用Scroll API从一组索引中获取所有文档,这些索引按created_at排序,以便稍后处理它们并将计算结果存储在不同的数据库中。这些索引中有大约1500万个文档,并按月分区(例如:user-events-2015-06user-events-2015-07等。

在我开始记录从Scroll API收到的每个批处理的最后一个元素的输出后,我注意到它们实际上没有正确排序。例如:

Took 0.65 seconds. Last event: 2017-01-11T08:20:10Z.
Took 0.49 seconds. Last event: 2017-01-11T10:54:26Z.
Took 0.33 seconds. Last event: 2017-01-14T02:04:50Z.
Took 0.39 seconds. Last event: 2017-01-12T08:06:41Z.
Took 0.36 seconds. Last event: 2017-01-12T20:03:04Z.
Took 1.0 seconds. Last event: 2017-01-16T02:01:43Z.
Took 0.35 seconds. Last event: 2017-01-13T11:03:59Z.
Took 1.16 seconds. Last event: 2017-01-17T08:05:50Z.
Took 1.29 seconds. Last event: 2017-01-27T12:28:54Z.

我正在使用Ruby来检索它,查询是:

client.search index: 'user-events-*', sort: 'created_at: asc', scroll: '3m', size: 5000

有没有办法不确定退回文件中的正确顺序?我对性能并不在乎,但准确性非常重要。我正在使用ES 2.4。

根据要求添加回复

{
    "_scroll_id" => "cXVlcnlUaGVuRmV0Y2g7MTU1OzE1NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE1NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE1ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE1OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwMTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwMjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwMzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwNDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwNTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwNjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwNzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwOTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxMTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxMjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxMzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxNDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxNTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxNjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxNzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxOTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyMTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyMjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyMzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyNDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyNTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyNjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyNzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyOTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzMTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzMjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzMzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzNDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzNTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzNjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzNzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzOTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwMTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwMjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwMzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwNDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwNTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwNjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwNzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwOTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMxMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzA7", "took" => 588, "timed_out" => false, "_shards" => {
        "total" => 155, "successful" => 155, "failed" => 0
    }, "hits" => {
        "total" => 17324219, "max_score" => nil, "hits" => [{
            "_index" => "user-events-2014-08",
            "_type" => "user-event",
            "_id" => "AVNLDBNkOoz7XgDuxTLp",
            "_score" => nil,
            "_source" => {
                [...] "created_at" => "2014-08-04T08:42:40Z"
            },
            "sort" => [1407141760000]
        }, {
            "_index" => "user-events-2014-08",
            "_type" => "user-event",
            "_id" => "AVNLC47tYXwn2KOKi-uF",
            "_score" => nil,
            "_source" => {
                [...] "created_at" => "2014-08-05T11:57:41Z"
            },
            "sort" => [1407239861000]
        }]
    }
}

0 个答案:

没有答案