Question

我有一个ELK部署来收集日志。现在，我需要提取所有包含一个特定字符串的日志。但是我遇到了一个有趣的问题，就是我在Kibana的开发工具中以及从elasticsearch python客户端获得了不同的输出。

这是Kibana中的查询：

GET app_web_log-20180827/_search
{
  "query": {
    "bool": {
      "must": [
        { "match_phrase": { "message":   "Failed to call Billing API Server" }}
      ],
      "filter": [
        { "term":  { "deployment": "app_instance1" }},
        { "term":  { "module": "test_module" }}, 
        { "range": { "@timestamp": { "gte": 1535266800000, "lt": 1535353200000 }}} 
      ]
    }
  },
  "size": 5
}

以下是开发工具的输出：

{
  "took": 556,
  "timed_out": false,
  "_shards": {
    "total": 175,
    "successful": 175,
    "skipped": 165,
    "failed": 0
  },
  "hits": {
    "total": 400,
    "max_score": 34.769733,
    "hits": [
      {
        "_index": "app_web_log-20180827",
        "_type": "doc",
        "_id": "FMkHeWUB_hBu7Tio4Llg",
        "_score": 34.769733,
        "_source": {
          "beat": {
            "version": "6.2.4",
            "name": "app-web001",
            "hostname": "app-web001"
          },
          "offset": 349461,
          "@timestamp": "2018-08-27T01:38:03.049Z",
          "source": "/apphome/app_instance1/logs/test_module.log",
          "message": "2018-08-27 01:37:59,661 [http-bio-8168-exec-8] ERROR [Billing APIClientImpl] Failed to call Billing API Server. Billing API Billing server response error, tranId:c95cede3a011d97fd9f3d661eb961cb8",
          "module": "test_module",
          "@version": "1",
          "deployment": "app_instance1"
        }
      },
....

但是当我查询时，请使用elasticsearch python客户端。它什么也没给我：

from elasticsearch import Elasticsearch
es = Elasticsearch([{'host': 'esserver', 'port': 9200, 'username': 'appuser', 'password': 'elastic'}])

body = {
  "query": { 
    "bool": { 
      "must": [
        { "match_phrase": { "message":   "Failed to call Billing API Server" }}
      ],
      "filter": [ 
        { "term":  { "deployment": "app_instance1" }},
        { "term":  { "module": "test_module" }}, 
        { "range": { "@timestamp": { "gte": 1535266800000, "lt": 1535353200000 }}} 
      ]
    }
  }
}
print body

page = es.search(index='app_web_log-20180827', doc_type='doc', body=body,
         scroll='2m', size=100)
sid = page['_scroll_id']
scroll_size = page['hits']['total']
while (scroll_size > 0):
    print "Scrolling..."
    page = es.scroll(scroll_id = sid, scroll = '2m')
    # Update the scroll ID
    sid = page['_scroll_id']
    # Get the number of results that we returned in the last scroll
    scroll_size = len(page['hits']['hits'])
    for m in page['hits']['hits']:
        msg = m['_source']['message']
        print msg

我什么也没有：

{'query': {'bool': {'filter': [{'term': {'deployment': 'app_instance1'}}, {'term': {'module': 'test_module'}}, {'range': {'@timestamp': {'lt': 1535353200000, 'gte': 1535266800000}}}], 'must': [{'match_phrase': {'message': 'Failed to call Billing API Server'}}]}}}
Scrolling...

我想知道代码中是否有错误？请帮助。谢谢

Answer 1

我建议您看看为您提供逻辑的'\n'助手（[0]）。

我认为，由于您只是在访问scan之后而不是之前访问页面，因此您不处理scroll API调用返回的匹配。您还将search设置为size，因此很有可能所有匹配都在您忽略的100变量的第一个值中。

0-https://elasticsearch-py.readthedocs.io/en/master/helpers.html#scan

Kibana开发工具和elasticsearch-py客户端获得不同的输出

1 个答案: