Question

我添加了这样的模板：

curl -X PUT "e.f.g.h:9200/_template/impression-template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["impression-%{+YYYY.MM.dd}"],
  "settings": {
    "number_of_shards": 2,
    "number_of_replicas": 2
  },
  "mappings": {
    "_doc": {
      "_source": {
        "enabled": false
      },
      "dynamic": false,
      "properties": {
        "message": {
          "type": "object",
          "properties": {
...

我有logstash实例，可从kafka读取事件，然后将事件写入ES。这是我的logstash配置：

input {
  kafka {
    topics => ["impression"]
    bootstrap_servers => "a.b.c.d:9092"
  }
}
filter {
  json {
    source => "message"
    target => "message"
  }
}
output {
    elasticsearch {
        hosts => ["e.f.g.h:9200"]
        index => "impression-%{+YYYY.MM.dd}"
        template_name => "impression-template"
    }
}

但是每天我都会得到5个分片和1个副本的索引（这是ES的默认配置）。我该如何解决，这样我才能得到2个副本和2个分片？

Answer 1

不确定是否可以将Time::Piece->gmtime添加为index_pattern，因为在创建my_index-%{+YYYY.MM.dd}和PUT my_index-2019.03.10时，由于无法识别，因此将具有空映射。我遇到了同样的问题，解决方法是将index_pattern设置为my_index-*并将年份后缀添加到看起来像my_index-2017, my_index-2018...

的索引中

{
  "my_index_template" : {
    "order" : 0,
    "index_patterns" : [
      "my_index-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "5",
        "number_of_replicas" : "1"
      }
    },...

我从时间戳字段(YYYY-MM-dd)中提取年份来生成year，并通过logstash将其添加到索引名称的末尾

grok {
      match => [
         "timestamp", "(?<index_year>%{YEAR})"
      ]
    }

    mutate {
        add_field => {
            "[@metadata][index_year]" => "%{index_year}"
        }
    }

    mutate {
        remove_field => [ "index_year", "@version" ]
    }
}
output{
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "my_index-%{[@metadata][index_year]}"
        document_id => "%{some_field}"
    }
}

完成logstash之后，我设法获得了具有5个分片，1个副本和正确映射的my_index-2017，my_index-2018和my_index-2019索引，我的模板。

副本和分片设置未在Elasticsearch模板中应用

1 个答案: