使用正则表达式的Kibana无法正常工作

时间：2018-10-20 18:23:06

标签： elasticsearch elastic-stack kibana-6

我对使用Kibana和Elastic搜索还很陌生，但是我可以使用它并且可以搜索字符串等。

但是我要搜索此正则表达式

threshold of 1000 ms: ([5-9],\d+|[1-9][0-9],\d+|[1-9][0-9][0-9],\d+)

使用https://regex101.com/和Notepad ++进行测试的情况。 但是无论我如何尝试，我都无法使其在Kibana中发挥作用。

我尝试将索引字段更改为关键字而不是文本，但是仍然无法正常工作。

甚至是一个简单的正则表达式

message:/192.168.99.[0-9]{3}/

无法正常工作。

有人可以建议一种使它正常工作的方法吗？真让我发疯。

我应该提到我正在使用Filebeat 6.4.2，Elasticsearch 6.4.2和Kibana 6.4.2

谢谢

映射

"mappings": {
  "doc": {
    "date_detection": false,
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "@version": {
        "type": "long"
      },
      "CorrelationId": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "InteractionNumber": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "RequestorId": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "alertType": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "app": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "beat": {
        "properties": {
          "hostname": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          },
          "version": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "host": {
        "properties": {
          "name": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "input": {
        "properties": {
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "level": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "logger_name": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "message": {
        "type": "keyword"
      },
      "nodeId": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "offset": {
        "type": "long"
      },
      "pegathread": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "prospector": {
        "properties": {
          "type": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      },
      "source": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "source_host": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "src-env": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "src-node": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "src-vm": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "stack": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "tenantid": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "thread_name": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "userid": {
        "type": "keyword",
        "ignore_above": 1024
      },
      "x-requested-with": {
        "type": "keyword",
        "ignore_above": 1024
      }
    }
  }
}

示例消息字段内容

2018-10-20 23:10:21,068 GMT*8*PEGA0001*8087*1000*8ce767fc2b32*NA*NA*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*Test.User*Case-CAS-FS-Work-Svc*Solution:01.03.01*00cb8b6febb234d359369e54a60a865f*Y*3*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*35*http-apr-8080-exec-26*STANDARD*com.pega.pegarules.session.internal.engineinterface.service.HttpAPI*192.168.99.100|192.168.99.1*Activity=Pega-UI-CommandPalette.pzGetPaletteOptions*Rule-Obj-Activity:pzGetPaletteOptions*PEGA-UI-COMMANDPALETTE PZGETPALETTEOPTIONS #20161123T194957.445 GMT Step: 2 Circum: 0*NA*****pxRDBIOElapsed=0.03;pxRDBIOCount=4;pxRunStreamCount=811;pxTotalReqCPU=2.81;pxRunModelCount=270;pxOutputBytes=584,268;pxRunWhenCount=1,904;pxDeclarativePageLoadElapsed=6.84;pxRulesExecuted=3,471;pxOtherCount=314;pxDBInputBytes=3,553,909;pxTotalReqTime=8.09;pxActivityCount=967;pxAlertCount=1;pxOtherFromCacheCount=66;pxInteractions=1;pxLegacyRuleAPIUsedCount=1;pxRuleCount=254;pxInputBytes=101;pxRuleIOElapsed=0.09;pxRulesUsed=4,262;pxDeclarativePageLoadCount=6;pxRuleFromCacheCount=254;pxOtherIOElapsed=0.99;pxTrackedPropertyChangesCount=106;pxOtherIOCount=255;*NA*NA*NA*NA*NA*pyActivity=Pega-UI-CommandPalette.pzGetPaletteOptions;primaryPageClass=Data-Portal-DesignerStudio;*HTTP interaction has exceeded the elapsed time alert threshold of 1000 ms: 8088 ms.*

将消息类型添加为字符串，这是_analyze结果：-

{
  "token": "threshold",
  "start_offset": 1276,
  "end_offset": 1285,
  "type": "<ALPHANUM>",
  "position": 128
},
{
  "token": "of",
  "start_offset": 1286,
  "end_offset": 1288,
  "type": "<ALPHANUM>",
  "position": 129
},
{
  "token": "1000",
  "start_offset": 1289,
  "end_offset": 1293,
  "type": "<NUM>",
  "position": 130
},
{
  "token": "ms",
  "start_offset": 1294,
  "end_offset": 1296,
  "type": "<ALPHANUM>",
  "position": 131
},
{
  "token": "8088",
  "start_offset": 1298,
  "end_offset": 1302,
  "type": "<NUM>",
  "position": 132
},
{
  "token": "ms",
  "start_offset": 1303,
  "end_offset": 1305,
  "type": "<ALPHANUM>",
  "position": 133
}

1 个答案:

答案 0 :(得分：1)

在版本6.2.4上测试

在下面的索引中添加了如下所示的映射

    PUT test
{
  "mappings": {
    "_doc": {
      "properties": {
        "message": {
          "type": "text"
        },
        "message2": {
          "type": "keyword"
        }
      }
    }
  }
}

将2个文档添加到索引如下

PUT test/_doc/1?refresh
{
  "message": "hellothere",
  "message2":"2018-10-20 23:10:21,068 GMT*8*PEGA0001*8087*1000*8ce767fc2b32*NA*NA*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*Test.User*Case-CAS-FS-Work-Svc*Solution:01.03.01*00cb8b6febb234d359369e54a60a865f*Y*3*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*35*http-apr-8080-exec-26*STANDARD*com.pega.pegarules.session.internal.engineinterface.service.HttpAPI*192.168.99.100|192.168.99.1*Activity=Pega-UI-CommandPalette.pzGetPaletteOptions*Rule-Obj-Activity:pzGetPaletteOptions*PEGA-UI-COMMANDPALETTE PZGETPALETTEOPTIONS #20161123T194957.445 GMT Step: 2 Circum: 0*NA*****pxRDBIOElapsed=0.03;pxRDBIOCount=4;pxRunStreamCount=811;pxTotalReqCPU=2.81;pxRunModelCount=270;pxOutputBytes=584,268;pxRunWhenCount=1,904;pxDeclarativePageLoadElapsed=6.84;pxRulesExecuted=3,471;pxOtherCount=314;pxDBInputBytes=3,553,909;pxTotalReqTime=8.09;pxActivityCount=967;pxAlertCount=1;pxOtherFromCacheCount=66;pxInteractions=1;pxLegacyRuleAPIUsedCount=1;pxRuleCount=254;pxInputBytes=101;pxRuleIOElapsed=0.09;pxRulesUsed=4,262;pxDeclarativePageLoadCount=6;pxRuleFromCacheCount=254;pxOtherIOElapsed=0.99;pxTrackedPropertyChangesCount=106;pxOtherIOCount=255;*NA*NA*NA*NA*NA*pyActivity=Pega-UI-CommandPalette.pzGetPaletteOptions;primaryPageClass=Data-Portal-DesignerStudio;*HTTP interaction has exceeded the elapsed time alert threshold of 1000 ms: 8088 ms.*" 
} 


PUT test/_doc/2?refresh
{
  "message": "2018-10-20 23:10:21,068 GMT*8*PEGA0001*8087*1000*8ce767fc2b32*NA*NA*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*Test.User*Case-CAS-FS-Work-Svc*Solution:01.03.01*00cb8b6febb234d359369e54a60a865f*Y*3*HKVZWM7PHSLMGR3ZXP4OEKEBG3DFFS30K*35*http-apr-8080-exec-26*STANDARD*com.pega.pegarules.session.internal.engineinterface.service.HttpAPI*192.168.99.100|192.168.99.1*Activity=Pega-UI-CommandPalette.pzGetPaletteOptions*Rule-Obj-Activity:pzGetPaletteOptions*PEGA-UI-COMMANDPALETTE PZGETPALETTEOPTIONS #20161123T194957.445 GMT Step: 2 Circum: 0*NA*****pxRDBIOElapsed=0.03;pxRDBIOCount=4;pxRunStreamCount=811;pxTotalReqCPU=2.81;pxRunModelCount=270;pxOutputBytes=584,268;pxRunWhenCount=1,904;pxDeclarativePageLoadElapsed=6.84;pxRulesExecuted=3,471;pxOtherCount=314;pxDBInputBytes=3,553,909;pxTotalReqTime=8.09;pxActivityCount=967;pxAlertCount=1;pxOtherFromCacheCount=66;pxInteractions=1;pxLegacyRuleAPIUsedCount=1;pxRuleCount=254;pxInputBytes=101;pxRuleIOElapsed=0.09;pxRulesUsed=4,262;pxDeclarativePageLoadCount=6;pxRuleFromCacheCount=254;pxOtherIOElapsed=0.99;pxTrackedPropertyChangesCount=106;pxOtherIOCount=255;*NA*NA*NA*NA*NA*pyActivity=Pega-UI-CommandPalette.pzGetPaletteOptions;primaryPageClass=Data-Portal-DesignerStudio;*HTTP interaction has exceeded the elapsed time alert threshold of 1000 ms: 8088 ms.*",
  "message2":"hellothere" 
}

搜索message2: /192.168.99.[0-9]{3}/结果为0个结果

在文档＃2中搜索message: /192.168.99.[0-9]{3}/结果

在doc＃1中搜索message2: /.*192.168.99.[0-9]{3}.*/结果

搜索message: /pegarules.session/结果为0个结果。

但是在文档＃1中搜索message: /.*pegarules.session.*/结果 因为倒排索引具有"token": "com.pega.pegarules.session.internal.engineinterface.service.httpapi"

在doc＃1中搜索message2: /.*pegarules.session.*/结果

因此，已归档的消息（类型text）已被标记化，正则表达式搜索通配符标记模式正在返回结果。

原样，不对message2字段（类型keyword）进行分析，而是将其原样放入倒排索引中。除非我们添加贪婪的量词（。*）

，否则正则表达式搜索192.168.99.[0-9]{3}之类的模式不会返回任何内容

Lucene正则表达式引擎不兼容Perl，但支持较小范围的运算符，因此它可能无法正常工作并匹配正则表达式等结果。

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html#regexp-syntax

相关问题
最新问题