Elasticsearch滚动api搜索“来自”

时间:2014-09-26 11:31:02

标签: php xml elasticsearch

我有一个脚本可以根据网址索引http://example.com/sitemap.index.xml生成站点地图,其中index是一个数字>0,用于定义每个块中应包含的结果。

$chunk = 10000;
$counter = 0;

$scroll = $es->search(array(
    "index" => "index",
    "type" => "type",
    "scroll" => "1m",
    "search_type" => "scan",
    "size" => 10,
    "from" => $chunk * ($index - 1)
));
$sid = $scroll['_scroll_id'];

while($counter < $chunk){
    $docs = $es->scroll(array(
        "scroll_id" => $sid,
        "scroll" => "1m"
    ));
    $sid = $docs['_scroll_id'];
    $counter += count($docs['hits']['hits']);
}

// ...

现在每次访问http://example.com/sitemap.1.xmlhttp://example.com/sitemap.2.xml时,ES返回的结果都完全相同。它返回50个结果(每个分片10个),但似乎不计算from = 0from = 10000

我使用elasticsearch-php作为ES库。

有什么想法吗?

1 个答案:

答案 0 :(得分:-1)

在Java中,可以按照以下方式完成

QueryBuilder query = QueryBuilders.matchAllQuery();
SearchResponse scrollResp = Constants.client.prepareSearch(index)
        .setTypes(type).setSearchType(SearchType.SCAN)
        .setScroll(new TimeValue(600000)).setQuery(query)
        .setSize(500).execute().actionGet();
while (true) {
    scrollResp = Constants.client
            .prepareSearchScroll(scrollResp.getScrollId())
            .setScroll(new TimeValue(600000)).execute().actionGet();
    System.out.println("Record count :"
            + scrollResp.getHits().getHits().length);
    total = total + scrollResp.getHits().getHits().length;
    System.out.println("Total record count: " + total);
    for (SearchHit hit : scrollResp.getHits()) {
    //handle the hit
    }
    // Break condition: No hits are returned
    if (scrollResp.getHits().getHits().length == 0) {
        System.out.println("All records are fetched");
        break;
    }
}

希望它有所帮助。