Question

如果我们滚动特定大小的块，有没有办法使用python api设置检索的文档数量的上限。所以，让我们说我希望最多可以在2K的块中滚动100K文档，其中有超过10Mil的文档可用。

我已经实现了类似对象的计数器，但我想知道是否有更自然的解决方案。

es_query = {"query": {"function_score": {"functions": [{"random_score": {"seed": "1234"}}]}}}
es = Elasticsearch(ADDRESS, port=PORT)


result = es.search(
    index="INDEX", 
    doc_type="DOC_TYPE", 
    body=es_query,
    size=2000,
    scroll="1m")

data = []
for hit in result["hits"]["hits"]:
    for d in hit["_source"]["attributes"]["data_of_interest"]:
        data.append(d)
        do_something(*args)


scroll_id = result['_scroll_id']
scroll_size = result["hits"]["total"]

i = 0
while(scroll_size>0):
    if i % 10000 == 0:
        print("Scrolling ({})...".format(i))

    result = es.scroll(scroll_id=scroll_id, scroll="1m")
    scroll_id = result["_scroll_id"]
    scroll_size = len(result['hits']['hits'])

data = []
for hit in result["hits"]["hits"]:
    for d in hit["_source"]["attributes"]["data_of_interest"]:
        data.append(d)
        do_something(*args)

i += 1
if i == 100000:
    break

Answer 1

对我来说，如果你只想要前100K，你应该首先缩小查询范围。这将加快你的过程。例如，您可以在日期添加过滤器。

关于代码我不知道除了使用计数器以外的其他方式。我只是更正缩进并删除if语句以便于阅读。

es_query = {"query": {"function_score": {"functions": [{"random_score": {"seed": "1234"}}]}}}
es = Elasticsearch(ADDRESS, port=PORT)


result = es.search(
    index="INDEX", 
    doc_type="DOC_TYPE", 
    body=es_query,
    size=2000,
    scroll="1m")

data = []
for hit in result["hits"]["hits"]:
    for d in hit["_source"]["attributes"]["data_of_interest"]:
        data.append(d)
        do_something(*args)

scroll_id = result['_scroll_id']
scroll_size = result["hits"]["total"]

i = 0
while(scroll_size > 0 & i < 100000):

    print("Scrolling ({})...".format(i))

    result = es.scroll(scroll_id=scroll_id, scroll="1m")
    scroll_id = result["_scroll_id"]
    scroll_size = len(result['hits']['hits'])

    # data = [] why redefining the list ? 
    for hit in result["hits"]["hits"]:
        for d in hit["_source"]["attributes"]["data_of_interest"]:
            data.append(d)
            do_something(*args)
    i ++

Elasticsearch滚动上限 - python api

1 个答案: