删除elasticsearch中的旧索引

时间：2015-10-30 06:33:54

标签： elasticsearch logstash elasticsearch-plugin

我的许多日志都以logstash-Year-Week格式编制索引。那就是如果我想删除超过几周的索引，我怎样才能在elasticsearch中实现这一点。有一种简单，无缝的方法吗？

10 个答案:

答案 0 :(得分：25)

策展人将是一个理想的比赛。 您可以在此处找到相关链接 - https://github.com/elastic/curator

下面的命令应该可以正常工作 - 

curator --host <IP> delete indices --older-than 30 --prefix "twitter-" --time-unit days  --timestring '%Y-%m-%d'

你可以在CRON中保留偶尔删除索引的内容。

您可以在此处找到一些示例和文档 - https://www.elastic.co/guide/en/elasticsearch/client/curator/current/examples.html

答案 1 :(得分：17)

如果您使用的是elasticsearch版本5.x，则需要安装curator版本4.x. 您可以从documentation

中查看版本兼容性和安装步骤

安装完毕。然后只需运行命令

curator --config path/config_file.yml [--dry-run] path/action_file.yml

Curator提供了一个干运行标志来输出Curator会执行的操作。输出将在您在config.yml文件中定义的日志文件中。如果没有在config_file.yml中定义的日志键，那么currator将输出到控制台。要删除索引，请运行上面的命令而不使用--dry-run flag

配置文件config_file.yml是

---
client:
  hosts:
   - 127.0.0.1
  port: 9200
logging:
  loglevel: INFO
  logfile: "/root/curator/logs/actions.log"
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

动作文件action_file.yml是

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

如果您想自动删除索引每周，每月等。然后编写像

这样的bash脚本 
#!/bin/bash
# Script to delete the log event indices of the elasticsearch weekly

#This will delete the indices of the last 7 days
curator --config /path/config_file.yml /path/action_file.yml

将shell脚本放在其中一个文件夹中：/etc/cron.daily, /etc/cron.hourly, /etc/cron.monthly or /etc/cron.weekly，您的工作已完成。

注意：确保在配置和操作文件中使用正确的缩进。否则它将无法工作。

答案 2 :(得分：12)

我使用bash脚本，只需使用您要保留的＃天数更改30 

#!/bin/bash

# Zero padded days using %d instead of %e
DAYSAGO=`date --date="30 days ago" +%Y%m%d`
ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v | egrep logstash`

echo
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:"
echo

echo "$ALLLINES" | while read LINE
do
  FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' ` 
  if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
  then
    TODELETE=`echo $LINE | awk '{ print $3 }'`
    echo "http://127.0.0.1:9200/$TODELETE"
  fi
done

echo
echo -n "if this make sence, Y to continue N to exit [Y/N]:"
read INPUT
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]
then
  echo "$ALLLINES" | while read LINE
  do
    FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | awk -F'-' '{ print $2 }' | sed 's/\.//g' `
    if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
    then
      TODELETE=`echo $LINE | awk '{ print $3 }'`
      /usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE
      sleep 1
      fi
  done
else 
  echo SCRIPT CLOSED BY USER, BYE ...
  echo
  exit
fi

答案 3 :(得分：6)

查看Curator，这是专门为此类用例开发的工具。

示例命令，用于文档：

curator --host 10.0.0.2 delete indices --older-than 30 --time-unit days \
   --timestring '%Y.%m.%d'

答案 4 :(得分：1)

您可以使用curl 

 curl -X DELETE http://localhost:9200/filebeat-$(date +"%Y.%m.%d" -d "last Month")

这必须将此命令添加到xxx.sh，然后您可以创建crontab。 crontab -e 

00 00 * * * /etc/elasticsearch/xxx.sh

此cron每天晚上12点运行，它将删除旧日志。

答案 5 :(得分：1)

从Elasticsearch 6.6开始，Index Lifecycle Management随基本（免费）版本的Elasticsearch一起提供，并实现了Curator以前的功能，但以一种更为优美的方式实现了。

未经马丁·埃因霍夫（MartinEhrnhöfer）出色而简洁的blog post的许可，复制了以下步骤。

假设（复制副本之前提示）：

  • 您可以在http://elasticsearch:9200上访问您的Elasticsearch服务器
  • 您希望在三十天（30d）后清除索引
  • 您的策略名称将创建为cleanup_policy
  • 您的文件索引索引名称以filebeat-开头
  • 您的logstash索引名称以logstash-开头

1。创建一个在一个月后删除索引的策略

curl -X PUT "http://elasticsearch:9200/_ilm/policy/cleanup_policy?pretty" \
     -H 'Content-Type: application/json' \
     -d '{
      "policy": {                       
        "phases": {
          "hot": {                      
            "actions": {}
          },
          "delete": {
            "min_age": "30d",           
            "actions": { "delete": {} }
          }
        }
      }
    }'

2。将此政策应用于所有现有的文件拍和logstash索引

curl -X PUT "http://elasticsearch:9200/logstash-*/_settings?pretty" \
     -H 'Content-Type: application/json' \
     -d '{ "lifecycle.name": "cleanup_policy" }'
curl -X PUT "http://elasticsearch:9200/filebeat-*/_settings?pretty" \
     -H 'Content-Type: application/json' \
     -d '{ "lifecycle.name": "cleanup_policy" }'

3。创建一个模板，将该策略应用于新的文件拍和日志记录索引

curl -X PUT "http://elasticsearch:9200/_template/logging_policy_template?pretty" \
     -H 'Content-Type: application/json' \
     -d '{
      "index_patterns": ["filebeat-*", "logstash-*"],                 
      "settings": { "index.lifecycle.name": "cleanup_policy" }
    }'

答案 6 :(得分：0)

yanb（又一个bash）

#!/bin/bash
searchIndex=logstash-monitor
elastic_url=logging.core.k94.kvk.nl
elastic_port=9200

date2stamp () {
    date --utc --date "$1" +%s
}

dateDiff (){
    case $1 in
        -s)   sec=1;      shift;;
        -m)   sec=60;     shift;;
        -h)   sec=3600;   shift;;
        -d)   sec=86400;  shift;;
        *)    sec=86400;;
    esac
    dte1=$(date2stamp $1)
    dte2=$(date2stamp $2)
    diffSec=$((dte2-dte1))
    if ((diffSec < 0)); then abs=-1; else abs=1; fi
    echo $((diffSec/sec*abs))
}

for index in $(curl -s "${elastic_url}:${elastic_port}/_cat/indices?v" |     grep -E " ${searchIndex}-20[0-9][0-9]\.[0-1][0-9]\.[0-3][0-9]" | awk '{     print $3 }'); do
  date=$(echo ${index: -10} | sed 's/\./-/g')
  cond=$(date +%Y-%m-%d)
  diff=$(dateDiff -d $date $cond)
  echo -n "${index} (${diff})"
  if [ $diff -gt 1 ]; then
    echo " / DELETE"
    # curl -XDELETE "${elastic_url}:${elastic_port}/${index}?pretty"
  else
    echo ""
  fi
done

答案 7 :(得分：0)

curator_cli delete_indices --filter_list '{"filtertype":"none"}'

将删除全部或过滤：

 --filter_list '[{"filtertype":"age","source":"creation_date","direction":"older","unit":"days","unit_count":13},{"filtertype":"pattern","kind":"prefix","value":"logstash"}]'

答案 8 :(得分：0)

就我而言，删除旧索引是强制性的，因为我已从5.X升级到7.5版本，

所以我按照简单的步骤清除索引。

rm -rf /var/lib/elasticsearch/nodes/0/indices/*

答案 9 :(得分：0)

策展人没有帮助我

现在 Curator 在使用以下命令运行时给我一个错误：

curator --config config_file.yml action_file.yml

错误：

Error: Elasticsearch version 7.9.1 incompatible with this version of Curator (5.2.0)

找不到与 Elasticsearch 7.9.1 兼容的 curator 版本，我无法升级或降级 elasticsearch 版本。因此，我使用@Alejandro 的答案并使用下面的脚本进行了操作。我稍微修改了脚本

脚本解决方案

#!/bin/bash

# Zero padded days using %d instead of %e
DAYSAGO=`date --date="30 days ago" +%Y%m%d`
ALLLINES=`/usr/bin/curl -s -XGET http://127.0.0.1:9200/_cat/indices?v`
# Just add -u <username>:<password> in curl statement if your elastic search is behind the credentials. Also, you can give an additional grep statement to filter out specific indexes

echo
echo "THIS IS WHAT SHOULD BE DELETED FOR ELK:"
echo

echo "$ALLLINES" | while read LINE
do
  FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | grep -Eo "[0-9]{4}.[0-9]{2}.[0-9]{2}" | sed 's/\.//g'`
  if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
  then
    TODELETE=`echo $LINE | awk '{ print $3 }'`
    echo "http://127.0.0.1:9200/$TODELETE"
  fi
done

echo
echo -n "Y to continue N to exit [Y/N]:"
read INPUT
if [ "$INPUT" == "Y" ] || [ "$INPUT" == "y" ] || [ "$INPUT" == "yes" ] || [ "$INPUT" == "YES" ]
then
  echo "$ALLLINES" | while read LINE
    do
    FORMATEDLINE=`echo $LINE | awk '{ print $3 }' | grep -Eo "[0-9]{4}.[0-9]{2}.[0-9]{2}" | sed 's/\.//g'`
    if [ "$FORMATEDLINE" -lt "$DAYSAGO" ]
    then
      TODELETE=`echo -n $LINE | awk '{ print $3 }'`
      /usr/bin/curl -XDELETE http://127.0.0.1:9200/$TODELETE
      sleep 1
      fi
  done
else
  echo SCRIPT CLOSED BY USER, BYE ...
  echo
  exit
fi
相关问题
最新问题