Question

我拥有以下文件的索引：

{name: 'Device1', type: 'start', 'eventTime': '2013-02-19 12:00:00'}
{name: 'Device2', type: 'start', 'eventTime': '2013-02-19 12:02:00'}
{name: 'Device1', type: 'stop', 'eventTime': '2013-02-19 12:45:00'}
{name: 'Device2', type: 'stop', 'eventTime': '2013-02-19 12:50:00'}

我想创建一个查询，它将为我返回一个新字段，其中eventTime的时间差异，设备名称和字段类型的问题。例如，它应该是：

{name: 'Device1', 'type': 'it really doesnt matter', eventTime: 'also doesnt matter', duration: '00:45:00'}
{name 'Device2', 'type': 'it really doesnt matter', eventTime: 'also doesnt matter', duration: '00:48:00'}

弹性搜索查询语言是否可行？

Answer 1

我认为目前每个文档的方式都不可能。如果您将文档存储为：

{name: 'Device1', startTime: '2013-02-19 12:00:00', endTime: '2013-02-19 12:45:00'}
{name: 'Device2', startTime: '2013-02-19 12:02:00', endTime: '2013-02-19 12:50:00'}

然后你可以返回script field这是时差。

{    
  "query" : {
    ...
  },
  "script_fields" : {
    "timedifference" : {
      "script" : "doc['endTime'].value - doc['startTime'].value"
    }
  }
}

Answer 2

您可以查看elapsed logstash过滤器。

对于您的用例，您必须使用 start_tag 标记起始文档，并使用 end_tag 停止文档。您可以使用“名称”作为唯一ID字段。

希望有所帮助！

Answer 3

搜索同样的事情，here我找到了一个很好的例子来回答这个问题，使用经过过滤的Ana说，并将它与聚合过滤器结合起来。

我们的想法是衡量每个阶段的时间，然后将事件之间的时间信息汇总到一个新事件中。

因此，按照链接中提供的示例，如果您有此日志：

2016-05-19T02:55:29.003 00e02f2f-32d5-9509-870a-f80e54dc8775 system1Enter
2016-05-19T02:55:29.200 00e02f2f-32d5-9509-870a-f80e54dc8775 system1Exit
2016-05-19T02:55:29.205 00e02f2f-32d5-9509-870a-f80e54dc8775 system2Enter
2016-05-19T02:55:29.453 00e02f2f-32d5-9509-870a-f80e54dc8775 system2Exit

我们定义了三个经过时间的过滤器（每个阶段为1，1-> 2和in2），然后是三个聚合过滤器，以便对所有定时信息求和。

filter {
  grok {
    match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{UUID:messageId} %{WORD:event}"]
    add_tag => [ "%{event}" ]
  }
  date {
    match => [ "timestamp", "ISO8601"]
  }
  # Measures the execution time of system1
  elapsed {
    unique_id_field => "messageId"
    start_tag => "system1Enter"
    end_tag => "system1Exit"
    new_event_on_match => true
    add_tag => ["in1"]
  }
  # Measures the execution time of system2
  elapsed {
    unique_id_field => "messageId"
    start_tag => "system2Enter"
    end_tag => "system2Exit"
    new_event_on_match => true
    add_tag => ["in2"]
  }
  # Measures the time between system1 and system2
  elapsed {
    unique_id_field => "messageId"
    start_tag => "system1Exit"
    end_tag => "system2Enter"
    new_event_on_match => true
    add_tag => ["1->2"]
  }
  # Records the execution time of system1
  if "in1" in [tags] and "elapsed" in [tags] {
    aggregate {
      task_id => "%{messageId}"
      code => "map['report'] = [(event['elapsed_time']*1000).to_i]"
      map_action => "create"
    }
  }
  # Records the time between system1 and system2
  if "1->2" in [tags] and "elapsed" in [tags] {
    aggregate {
      task_id => "%{messageId}"
      code => "map['report'] << (event['elapsed_time']*1000).to_i"
      map_action => "update"
    }
  }
  # Records the execution time of system2
  if "in2" in [tags] and "elapsed" in [tags] {
    aggregate {
      task_id => "%{messageId}"
      code => "map['report'] << (event['elapsed_time']*1000).to_i; event['report'] = map['report'].join(':')"
      map_action => "update"
      end_of_task => true
    }
  }
}

在前两个事件之后，您将收到一个新事件，显示已在system1中花费了197毫秒：

{
                 "@timestamp" => "2016-05-21T04:20:51.731Z",
                       "tags" => [ "elapsed", "elapsed_match", "in1" ],
               "elapsed_time" => 0.197,
                  "messageId" => "00e02f2f-32d5-9509-870a-f80e54dc8775",
    "elapsed_timestamp_start" => "2016-05-19T00:55:29.003Z"
}

同样，所有学分都转到原答案的主题：）

Answer 4

您可以使用scripted_metric来获取持续时间。

{
  "size": 0,
  "aggs": {
    "d_ids": {
      "terms": {
        "field": "name.keyword",
        "size": 10
      },
          "aggs": {
            "duration": {
              "scripted_metric": { 
                "map_script": "if (doc.type.value == \"stop\") { params._agg.end = doc.eventTime.value; } else { params._agg.start = doc.eventTime.value; }",
                "reduce_script": "long start = 0; long end = 0; for(h in params._aggs) { if(h.start != null) { start = h.start; } if (h.end != null) { end = h.end; } } return (end - start);"
              }
            }
          }
    }
  }
}

以下是上述查询的回复：

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 4,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "d_ids": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "Device1",
          "doc_count": 2,
          "duration": {
            "value": 2700000
          }
        },
        {
          "key": "Device2",
          "doc_count": 2,
          "duration": {
            "value": 2880000
          }
        }
      ]
    }
  }
}

持续时间以毫秒为单位

如何创建可以计算时差的查询？

4 个答案: