弹性搜索基于日期的嵌套查询过滤器未返回正确的结果

时间:2018-03-14 14:06:25

标签: elasticsearch

我有嵌套查询在哪里,我在过滤当前数据,然后使用日期 - 直方图聚合与每小时间隔聚合数据,但在日期 - 直方图输出中,它也返回前一天的数据。过滤器不起作用?

这是我的查询:

POST finalalertbrowser/_search?size=0
{
    "query": {
        "bool": {
            "must": [{
                "match_phrase": {
                    "projectId.keyword": "******************************88"
                }
            }],
            "filter": {
                "nested": {
                    "path": "errors",
                    "query": {
                        "bool": {
                            "filter": 
                            {
                              "range": {
                                    "errors.time": {
                                        "gte": "now/d",
                                        "lte": "now"
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    },
    "aggs": {
        "errorData": {
            "nested": {
                "path": "errors"
            },
            "aggs": {
                "errorMsg": {
                    "filter": {
                      "term": {
                            "errors.errMsg.keyword": "Uncaught TypeError: $.snapUpdate is not a function"
                        }

                    },
                    "aggs": {
                        "hourlyData": {
                            "date_histogram": {
                                "field": "errors.time",
                                "interval": "hour",
                                "time_zone": "+05:30"
                            }
                        }
                    }
                }
            }
        }
    }
}

并且查询的输出是:

"aggregations": {
    "errorData": {
      "doc_count": 89644,
      "errorMsg": {
        "doc_count": 1861,
        "hourlyData": {
          "buckets": [
            {
              "key_as_string": "2018-03-13T11:00:00.000+05:30",
              "key": 1520919000000,
              "doc_count": 3
            },
            {
              "key_as_string": "2018-03-13T12:00:00.000+05:30",
              "key": 1520922600000,
              "doc_count": 2
            },
            {
              "key_as_string": "2018-03-13T13:00:00.000+05:30",
              "key": 1520926200000,
              "doc_count": 2
            },
            {
              "key_as_string": "2018-03-13T14:00:00.000+05:30",
              "key": 1520929800000,
              "doc_count": 2
            },
            {
              "key_as_string": "2018-03-13T15:00:00.000+05:30",
              "key": 1520933400000,
              "doc_count": 4
            },
            {
              "key_as_string": "2018-03-13T16:00:00.000+05:30",
              "key": 1520937000000,
              "doc_count": 8
            },
            {
              "key_as_string": "2018-03-13T17:00:00.000+05:30",
              "key": 1520940600000,
              "doc_count": 6
            },
            {
              "key_as_string": "2018-03-13T18:00:00.000+05:30",
              "key": 1520944200000,
              "doc_count": 3
            },
            {
              "key_as_string": "2018-03-13T19:00:00.000+05:30",
              "key": 1520947800000,
              "doc_count": 1
            },
            {
              "key_as_string": "2018-03-13T20:00:00.000+05:30",
              "key": 1520951400000,
              "doc_count": 2
            },
            {
              "key_as_string": "2018-03-13T21:00:00.000+05:30",
              "key": 1520955000000,
              "doc_count": 4
            },
            {
              "key_as_string": "2018-03-13T22:00:00.000+05:30",
              "key": 1520958600000,
              "doc_count": 3
            },
            {
              "key_as_string": "2018-03-13T23:00:00.000+05:30",
              "key": 1520962200000,
              "doc_count": 2
            },
            {
              "key_as_string": "2018-03-14T00:00:00.000+05:30",
              "key": 1520965800000,
              "doc_count": 1
            },
            {
              "key_as_string": "2018-03-14T01:00:00.000+05:30",
              "key": 1520969400000,
              "doc_count": 2
            },
            {
              "key_as_string": "2018-03-14T02:00:00.000+05:30",
              "key": 1520973000000,
              "doc_count": 1
            },
            {
              "key_as_string": "2018-03-14T03:00:00.000+05:30",
              "key": 1520976600000,
              "doc_count": 1
            },
            {
              "key_as_string": "2018-03-14T04:00:00.000+05:30",
              "key": 1520980200000,
              "doc_count": 2
            },
            {
              "key_as_string": "2018-03-14T05:00:00.000+05:30",
              "key": 1520983800000,
              "doc_count": 2
            },
            {
              "key_as_string": "2018-03-14T11:00:00.000+05:30",
              "key": 1521005400000,
              "doc_count": 349
            },
            {
              "key_as_string": "2018-03-14T12:00:00.000+05:30",
              "key": 1521009000000,
              "doc_count": 300
            },
            {
              "key_as_string": "2018-03-14T13:00:00.000+05:30",
              "key": 1521012600000,
              "doc_count": 258
            },
            {
              "key_as_string": "2018-03-14T14:00:00.000+05:30",
              "key": 1521016200000,
              "doc_count": 247
            },
            {
              "key_as_string": "2018-03-14T15:00:00.000+05:30",
              "key": 1521019800000,
              "doc_count": 144
            },
            {
              "key_as_string": "2018-03-14T16:00:00.000+05:30",
              "key": 1521023400000,
              "doc_count": 63
            },
            {
              "key_as_string": "2018-03-14T17:00:00.000+05:30",
              "key": 1521027000000,
              "doc_count": 30
            }
          ]
        }
      }
    }
  }

我已经在2018年3月14日执行了查询,但查询从2018年3月13日开始输出。

以下是映射命令:

PUT myIndexName
{
 "mappings": {
"webbrowsererror": {
     "properties": {
       "errors": {
         "type": "nested" ,
         "properties": {
           "time":{"type":"date"}
           }
       }
     }
   }
}
}

及以下是索引中的示例记录:

_source": {
          "projectId": "******************",
          "sId": "bt82x3g8v1505001600027",
          "pId": "bt82x3g8v1505001600027.1",
          "pageURL": "***************************",
          "startTime": 1505001600027,
          "country": "unknown",
          "size": 2,
          "errors": [
            {
              "sid": "bt82x3g8v1505001600027",
              "pid": "bt82x3g8v1505001600027.1",
              "browser": "Googlebot",
              "time": 1505001600028,
              "errMsg": "Uncaught SyntaxError: Invalid regular expression: missing /",
              "url": "********************************",
              "lineNo": 161,
              "colNo": 54
            },
            {
              "sid": "bt82x3g8v1505001600027",
              "pid": "bt82x3g8v1505001600027.1",
              "browser": "Googlebot",
              "time": 1505001600058,
              "errMsg": "Uncaught Error: Syntax error, unrecognized expression: #!",
              "url": "************************************************************",
              "lineNo": 3,
              "colNo": 69
            }
          ]
        }




 "_source": {
          "projectId": "shaan-shaanstack-1-1517388493060",
          "sId": "bt82x3g8v1502496000027",
          "pId": "bt82x3g8v1502496000027.1",
          "startTime": 1502496000027,
          "country": "US",
          "size": 1,
          "errors": [
            {
              "sid": "bt82x3g8v1502496000027",
              "pid": "bt82x3g8v1502496000027.1",
              "browser": "Chrome Mobile",
              "time": 1502496000128,
              "errMsg": "Uncaught Error: Syntax error, unrecognized expression: #!",
              "url": "**************************************************",
              "lineNo": 2,
              "colNo": 69
            }
          ]
        }





"_source": {
          "projectId": null,
          "sId": "888888888888888",
          "pId": "bt82x3g8v1505001600027.1",
          "pageURL": "******************",
          "startTime": 1505001600027,
          "country": "unknown",
          "size": 2,
          "errors": [
            {
              "sid": "bt82x3g8v1505001600027",
              "pid": "bt82x3g8v1505001600027.1",
              "browser": "Googlebot",
              "time": 1505001600028,
              "errMsg": "Uncaught SyntaxError: Invalid regular expression: missing /",
              "url": "***********************************",
              "lineNo": 170,
              "colNo": 54
            },
            {
              "sid": "bt82x3g8v1505001600027",
              "pid": "bt82x3g8v1505001600027.1",
              "browser": "Googlebot",
              "time": 1505001600082,
              "errMsg": "Uncaught Error: Syntax error, unrecognized expression: #!",
              "url": "***********************************",
              "lineNo": 3,
              "colNo": 69
            }
          ]
        }

2 个答案:

答案 0 :(得分:1)

我相信您的查询存在一些问题:

  1. 对于projectID,您希望使用terms查询来获得完全匹配
  2. 您的nested查询应在bool/must条款
  3. 范围内

    尝试一下(注意:我排除了聚合部分):

    {
      "sort": [
        {
          "errors.time": {
            "order": "asc"
          }
        }
      ], 
      "query": {
        "bool": {
          "must": [
            {
              "term": {
                "projectId.keyword": {
                  "value": "******************************88"
                }
              }
            },
            {
              "nested": {
                "path": "errors",
                "query": {
                  "range": {
                    "errors.time": {
                      "gte": "now/d",
                      "lte": "now"
                    }
                  }
                }
              }
            }
          ]
        }
      }
    }
    

    验证您的查询只返回正确的数据后,您就可以在汇总中重新添加

答案 1 :(得分:1)

您需要将嵌套字段视为父记录的一部分。让我们看下面的例子,我插入一个有2个嵌套属性的记录,一个时间为“2018-01-01T00:00:00Z”,另一个时间为“2018-01-02T00:00:00Z”

插入命令:

POST jaytest/webbrowsererror
{
  "projectId": "******************",
  "sId": "bt82x3g8v1505001600027",
  "pId": "bt82x3g8v1505001600027.1",
  "pageURL": "***************************",
  "startTime": 1505001600027,
  "country": "unknown",
  "size": 2,
  "errors": [
    {
      "sid": "bt82x3g8v1505001600027",
      "pid": "bt82x3g8v1505001600027.1",
      "browser": "Googlebot",
      "time": "2018-01-01T00:00:00Z",
      "errMsg": "Uncaught SyntaxError: Invalid regular expression: missing /",
      "url": "********************************",
      "lineNo": 161,
      "colNo": 54
    },
    {
      "sid": "bt82x3g8v1505001600027",
      "pid": "bt82x3g8v1505001600027.1",
      "browser": "Googlebot",
      "time": "2018-01-02T00:00:00Z",
      "errMsg": "Uncaught Error: Syntax error, unrecognized expression: #!",
      "url": "************************************************************",
      "lineNo": 3,
      "colNo": 69
    }
  ]
}

现在,我可以查询并说“只返回记录,其中errors.time是> =”2018-01-02T00:00:00Z“

GET jaytest/webbrowsererror/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "nested": {
            "path": "errors",
            "query": {
              "range": {
                "errors.time": {
                  "gte": "2018-01-02T00:00:00Z"
                }
              }
            }
          }
        }
      ]
    }
  }
}

当您运行该查询时,您会注意到它返回我插入的单个父记录,但同时包含两个嵌套的“错误”。那是因为你在查询父记录。

要按照你想要的方式对数据进行切片我认为正确的方法是摆脱嵌套的“错误”字段,而是将每个错误索引为它自己的文档(而不是父文档的嵌套子项)。 / p>