Question

我正在尝试将elasticsearch与logstash结合使用，并将其用于存储我的exim日志。

特别是，我想从日志文件中提取消息ID字段以简化搜索：

  grok {
    match => [
        "@message", "%{DATE} %{TIME} %{HOSTNAME:msgid} %{GREEDYDATA:details}"
    ]
  }
  mutate {
    gsub => [
        "msgid","[\\\:-]",""
    ]
  }

由于elasticsearch试图将每个字符串解析为Date，其中包含符号：/或 - ，我用mutate过滤器替换它们。

不幸的是，即使过滤的msg id也不被elasticsearch接受，问题是为什么？

    [2013-12-24 21:32:32,823][DEBUG][action.bulk              ] [Piledriver] 
    [logstash 2013.12.24][0] failed to execute bulk item (index) index 
    {[logstash-2013.12.24][exim][_7-j53yZRzmARuYsJEfgIA], 
    source[{"message":"<22>Dec 24 21:32:31 host exim[15691]: 
          2013-12-24 21:32:31 1VvWmN-000453-Fz Completed",
          "@version":"1",
          "@timestamp":"2013-12-24T21:32:31.000+03:00",
          "type":"exim",
          "host":"192.168.169.228",
          "syslog_pri":"22",
          "syslog_program":"exim",
          "syslog_pid":"15691",
          "received_at":"2013-12-24 18:32:31 UTC",
          "received_from":"192.168.169.228",
          "syslog_severity_code":6,
          "syslog_facility_code":2,
          "syslog_facility":"mail",
          "syslog_severity":"informational",
          "@source_host":"host",
          "@message":"2013-12-24 21:32:31 1VvWmN-000453-Fz Completed",
          "msgid":"1VvWmN000453Fz"}]}
    org.elasticsearch.index.mapper.MapperParsingException: failed to parse [msgid]
        at   org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:401)
        at org.elasticsearch.index.mapper.object.ObjectMapper.serializeValue(ObjectMapper.java:613)
        at org.elasticsearch.index.mapper.object.ObjectMapper.parse(ObjectMapper.java:466)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:516)
        at org.elasticsearch.index.mapper.DocumentMapper.parse(DocumentMapper.java:460)
        at org.elasticsearch.index.shard.service.InternalIndexShard.prepareCreate(InternalIndexShard.java:353)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardIndexOperation(TransportShardBulkAction.java:402)
        at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:156)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:556)
        at org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:426)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
Caused by: org.elasticsearch.index.mapper.MapperParsingException: failed to parse date field [1VvWmN000453Fz], tried both date format [dateOptionalTime], and timestamp number with locale []
        at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:487)
        at org.elasticsearch.index.mapper.core.DateFieldMapper.innerParseCreateField(DateFieldMapper.java:424)
        at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:194)
        at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(AbstractFieldMapper.java:390)
        ... 12 more
Caused by: java.lang.IllegalArgumentException: Invalid format: "1VvWmN000453Fz" is malformed at "VvWmN000453Fz"
        at org.elasticsearch.common.joda.time.format.DateTimeFormatter.parseMillis(DateTimeFormatter.java:754)
        at org.elasticsearch.index.mapper.core.DateFieldMapper.parseStringValue(DateFieldMapper.java:481)
        ... 15 more

Answer 1

如果您已将索引应用于索引，请分享您的elasticsearch Mapping。如果没有，则共享创建索引后创建的默认映射。

你也可以尝试将默认映射作为String给msgid

Elasticsearch将字符串视为日期

1 个答案: