Question

我正在尝试使用Logstash将ES的_id元数据字段输出到CSV文件中。

{
  "_index": "data",
  "_type": "default",
  "_id": "vANfNGYB9XD0VZRJUFfy",
  "_version": 1,
  "_score": null,
  "_source": {
    "vulnid": "CVE-2018-1000060",
    "product": [],
    "year": "2018",
    "month": "02",
    "day": "09",
    "hour": "23",
    "minute": "29",
    "published": "2018-02-09T18:29:02.213-05:00",
  },
  "sort": [
    1538424651203
  ]
}

我的logstash输出过滤器是：

output { csv {  fields => [ "_id", "vulnid", "published"]  path =>
"/tmp/export.%{+YYYY-MM-dd-hh-mm}.csv" } }

我得到输出：

,CVE-2018-1000060,2018-02-09T18:29:02.213-05:00

但是我想得到：

vANfNGYB9XD0VZRJUFfy,CVE-2018-1000060,2018-02-09T18:29:02.213-05:00

如何将元数据_id输出到csv文件中？是否指定“ _id”或“ @_id”或“ @id”之类的字段都没有关系。

Answer 1

查询ES时，必须启用docinfo => true。默认情况下为false。

input {
 elasticsearch {
  hosts => [ your hosts ]
  index => "ti"
  query => '{your query}'
  size => 1000
  scroll => "1s"
  docinfo => true
  schedule => "14 * * * *"
 }
}

Answer 2

好日志日志无法从您的输入中获取“ _id”字段，因为您必须未将选项 docinfo 设置为true。

docinfo有助于包含Elasticsearch文档信息，例如索引，类型_id等。请在此处查看更多信息https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html#plugins-inputs-elasticsearch-docinfo

将输入插件用作

input {
  elasticsearch {
    hosts => "hostname"
    index => "yourIndex"
    query => '{ "query": { "query_string": { "query": "*" } } }' //optional
    size => 500 //optional
    scroll => "5m" //optional
    docinfo => true
  }
}

使用Logstash输出csv插件从ElasticSearch输出文档元数据

2 个答案: