我正在尝试使用Scroll API从一组索引中获取所有文档,这些索引按created_at
排序,以便稍后处理它们并将计算结果存储在不同的数据库中。这些索引中有大约1500万个文档,并按月分区(例如:user-events-2015-06
,user-events-2015-07
等。
在我开始记录从Scroll API收到的每个批处理的最后一个元素的输出后,我注意到它们实际上没有正确排序。例如:
Took 0.65 seconds. Last event: 2017-01-11T08:20:10Z.
Took 0.49 seconds. Last event: 2017-01-11T10:54:26Z.
Took 0.33 seconds. Last event: 2017-01-14T02:04:50Z.
Took 0.39 seconds. Last event: 2017-01-12T08:06:41Z.
Took 0.36 seconds. Last event: 2017-01-12T20:03:04Z.
Took 1.0 seconds. Last event: 2017-01-16T02:01:43Z.
Took 0.35 seconds. Last event: 2017-01-13T11:03:59Z.
Took 1.16 seconds. Last event: 2017-01-17T08:05:50Z.
Took 1.29 seconds. Last event: 2017-01-27T12:28:54Z.
我正在使用Ruby来检索它,查询是:
client.search index: 'user-events-*', sort: 'created_at: asc', scroll: '3m', size: 5000
有没有办法不确定退回文件中的正确顺序?我对性能并不在乎,但准确性非常重要。我正在使用ES 2.4。
根据要求添加回复
{
"_scroll_id" => "cXVlcnlUaGVuRmV0Y2g7MTU1OzE1NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE1NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE1ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE1OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE2OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE3OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE4OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzE5OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwMTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwMjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwMzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwNDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwNTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwNjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwNzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIwOTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxMTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxMjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxMzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxNDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxNTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxNjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxNzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIxOTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyMTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyMjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyMzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyNDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyNTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyNjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyNzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIyOTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzMTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzMjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzMzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzNDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzNTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzNjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzNzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzIzOTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI0OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI1OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI2OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI3OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI4OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5MDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5MTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5MjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5MzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5NDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5NTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5NjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5NzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5ODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzI5OTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwMTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwMjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwMzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwNDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwNTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwNjpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwNzpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwODpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMwOTpKcXB6WHlncFJyeURfbzZlT3dFczZnOzMxMDpKcXB6WHlncFJyeURfbzZlT3dFczZnOzA7", "took" => 588, "timed_out" => false, "_shards" => {
"total" => 155, "successful" => 155, "failed" => 0
}, "hits" => {
"total" => 17324219, "max_score" => nil, "hits" => [{
"_index" => "user-events-2014-08",
"_type" => "user-event",
"_id" => "AVNLDBNkOoz7XgDuxTLp",
"_score" => nil,
"_source" => {
[...] "created_at" => "2014-08-04T08:42:40Z"
},
"sort" => [1407141760000]
}, {
"_index" => "user-events-2014-08",
"_type" => "user-event",
"_id" => "AVNLC47tYXwn2KOKi-uF",
"_score" => nil,
"_source" => {
[...] "created_at" => "2014-08-05T11:57:41Z"
},
"sort" => [1407239861000]
}]
}
}