重新索引特定模式的所有索引的弹性数据

时间：2018-05-22 10:45:23

标签： linux shell elasticsearch sh

我有多个弹性指数。索引的名称具有特定格式。以下是我的指数的一个例子：

  • abc_2ab99742-94d2-43f8-A582-ce10a0f031dc;

  • abc_e8241182-1a40-410b-a95d-c883472444f4;

现在我需要重新索引这些索引中的所有数据。为此，我编写了一个shell脚本，它正在执行以下操作。

1. Loop for all indices
  1.1 create a temporary index like abc_2ab99742-94d2-43f8-a582-ce10a0f031dc_tmp.
  1.2 reindix all the data from the original index to temp.
  1.3 delete and re-create the original index.
  1.4 reindex the data from temp to original index.
  1.5 delete the temporary index.

以下是shell脚本，我也写过。

#!/bin/bash

ES_HOST="localhost"
ES_PORT="9200"
TMP="_tmp"

indices=$(curl -s "http://${ES_HOST}:${ES_PORT}/_cat/indices/abc_*?h=index" | egrep 'abc_[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{8}*')

# do for all abc elastic indices
for index in $indices
do
    echo "Reindex process starting for index: $index"
    tmp_index=$index${TMP}
    output=$(curl -X PUT "http://${ES_HOST}:${ES_PORT}/$tmp_index" -H 'Content-Type: application/json' -d'
    {
        "settings" : {
            "index" : {
                "number_of_shards" : 16,
                "number_of_replicas" : 1
            }
        }
    }')
    echo "Temporary index: $tmp_index created with output: $output"
    echo "Starting reindexing elastic data from original index:$index to temporary index:$tmp_index"
    output=$(curl -X POST "http://${ES_HOST}:${ES_PORT}/_reindex" -H 'Content-Type: application/json' -d'
    {
      "source": {
        "index": '"$index"'
      },
      "dest": {
        "index": '"$tmp_index"'
      }
    }
    ')
    echo "Reindexing completed from original index:$index to temporary index:$tmp_index with output: $output"
    echo "Deleting $index"
    output=$(curl -X DELETE "http://${ES_HOST}:${ES_PORT}/$index")
    echo "$index deleted with status: $output"
    echo "Creating index: $index"
    output=$(curl -X PUT "http://${ES_HOST}:${ES_PORT}/$index" -H 'Content-Type: application/json' -d'
    {
        "settings" : {
            "index" : {
                "number_of_shards" : 16,
                "number_of_replicas" : 1
            }
        }
    }')
    echo "Index: $index creation status: $output"
    echo "Starting reindexing elastic data from temporary index:$tmp_index to original index:$index"
    output=$(curl -X POST "http://${ES_HOST}:${ES_PORT}/_reindex" -H 'Content-Type: application/json' -d'
    {
      "source": {
        "index": '"$tmp_index"'
      },
      "dest": {
        "index": '"$index"'
      }
    }
    ')
    echo "Reindexing completed from temporary index:$tmp_index to original index:$index with output: $output"
    echo "Deleting $tmp_index"
    output=$(curl -X DELETE "http://${ES_HOST}:${ES_PORT}/$tmp_index")
    echo "$tmp_index deleted with status: $output"
done

但是我在reindex命令中遇到异常。以下是例外

Reindexing completed from original index:abc_58b888be-a90f-e3be-838d-88877aee572c to temporary index:abc_58b888be-a90f-e3be-838d-88877aee572c_tmp with output: {"error":{"root_cause":[{"type":"parsing_exception","reason":"[reindex] failed to parse field [source]","line":4,"col":9}],"type":"parsing_exception","reason":"[reindex] failed to parse field [source]","line":4,"col":9,"caused_by":{"type":"json_parse_exception","reason":"Unrecognized token 'abc_58b888be': was expecting ('true', 'false' or 'null')\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@39913ba; line: 4, column: 31]"}},"status":400}

任何人都可以帮助我，因为我在shell脚本方面不是很好。

2 个答案:

答案 0 :(得分：0)

问题在于您的shell脚本，请检查以下部分; 

output=$(curl -X POST "http://${ES_HOST}:${ES_PORT}/_reindex" -H 'Content-Type: application/json' -d'
    {
      "source": {
        "index": '"$index"'
      },
      "dest": {
        "index": '"$tmp_index"'
      }
    }
    ')

这里你假设发布一个json，但是json无效，改变如下，那么你的脚本就可以了：

output=$(curl -XPOST "http://${ES_HOST}:${ES_PORT}/_reindex" -H 'Content-Type: application/json' -d'
    {
      "source": {
        "index": "'"$index"'"
      },
      "dest": {
        "index": "'"$tmp_index"'"
      }
    }
    ')



"index": "'"$index"'"

根据json格式

创建一个有效的键值对

答案 1 :(得分：0)

这是重新索引 elasticsearch 索引的相同修改脚本。

#!/bin/bash

#ES_HOST="localhost"
#ES_PORT="9200"
TMP="_v2"
echo "****  Script Execution Started ****"
echo " "
echo "**** Fetching List of Indices Elasticsearch Reindexing ****"
curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc' 

echo " "
sleep 5

indices_list=$(curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc' | awk '{print $3}' | sed -n '1!p')
#indices=$(curl -s "http://${ES_HOST}:${ES_PORT}/_cat/indices/abc_*?h=index" | egrep 'abc_[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{8}*')

# do for all abc elastic indices
count=0
echo "$indices_list"
for index in $indices_list
do
    echo " "
    echo "**** Index No: $count ****"
    echo "**** Present Index we are iterating:  ${index} ****"
    count=`expr $count + 1`
    echo " "
    echo " "
    echo "Reindex process starting for index: $index"
    tmp_index=$index${TMP}
    output=$(curl -s -X PUT "http://localhost:9200/$tmp_index" -H 'Content-Type: application/json' -d'
    {
        "settings" : {
            "index" : {
                "number_of_shards" : 5,
                "number_of_replicas" : 1
            }
        }
    }')
    echo " "
    echo "Temporary index: $tmp_index created with output: $output"
    echo "Starting reindexing elastic data from original index: $index to temporary index: $tmp_index"
    output=$(curl -s -X POST "http://localhost:9200/_reindex" -H 'Content-Type: application/json' -d'
    {
      "source": {
        "index": "'$index'"
      },
      "dest": {
        "index": "'$tmp_index'"
      }
    }
    ')
    echo " "
    echo "Reindexing completed from original index: $index to temporary index: $tmp_index with output: $output"
    echo " "
    echo "Deleting $index"
    output=$(curl -s -X DELETE "http://localhost:9200/$index")
    echo "$index deleted with status: $output"
    echo " "
    echo "Creating index: $index"
    output=$(curl -s -X PUT "http://localhost:9200/$index" -H 'Content-Type: application/json' -d'
    {
        "settings" : {
            "index" : {
                "number_of_shards" : 5,
                "number_of_replicas" : 1
            }
        }
    }')
    echo " "
    echo "Index: $index creation status: $output"
    echo " "
    echo "Starting reindexing elastic data from temporary index: $tmp_index to original index: $index"
    output=$(curl -s -X POST "http://localhost:9200/_reindex" -H 'Content-Type: application/json' -d'
    {
      "source": {
        "index": "'$tmp_index'"
      },
      "dest": {
        "index": "'$index'"
      }
    }
    ')
    echo " "
    echo "Reindexing completed from temporary index:$tmp_index to original index:$index with output: $output"
    echo " "
    echo "Deleting $tmp_index"
    output=$(curl -s -X DELETE "http://localhost:9200/$tmp_index")
    echo "$tmp_index deleted with status: $output"
    echo " "
done

echo " "
sleep 5
echo "**** Fetching List of Indices After Elasticsearch Reindexing ****"
curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc' 
echo " "
sleep 5
echo " "
echo "**** Original indices list ****"
echo "$indices_list"
echo "**** No. of indices in original list: $count ****"
echo " "
count1=0
MIndices_list=$(curl -s -X GET 'http://localhost:9200/_cat/indices/%2A?v=&s=index:desc' | awk '{print $3}' | sed -n '1!p')
echo " "
echo "**** Modified indices list ****"
echo "$MIndices_list"
for j in $MIndices_list
do
     count1=`expr $count1 + 1`
done
echo " "
echo "**** No. of indices in modified list: $count1 ****"
echo "****  Script Execution Ended ****"
相关问题
最新问题