Question

我有一个AWS EC2实例，它有一个运行elasticsearch的docker容器。每天，某些流程都会将新文档推送到elasticsearch中。我希望在文档进入elasticsearch后的第二天将这些文档推送到S3存储桶。

我在网上找到了这个解决方案：

https://github.com/AckeeDevOps/elasticsearch-backup-to-s3

我是elasticsearch，docker和aws的新手，所以git存储库中提供的步骤不够详细，我不知道该怎么做，所以我想知道是否有人可以验证我对我需要采取的步骤的理解，下面：

0）克隆上面的git存储库并设置以下参数：

ELASTICSEARCH_URL url with port where your elasticsearch runs, for example localhost:9200
S3_URL contains address in S3 where to store backups
    bucket-name/directory
S3_ACCESS_KEY
S3_SECRET_KEY
CRON_SCHEDULE cron schedule string, default '0 2 * * *'

1）在我的EC2实例上，执行

docker build https://github.com/AckeeDevOps/elasticsearch-backup-to-s3

2）然后，我需要更改正在运行的现有docker容器以具有其他选项

/var/backup/elasticsearch -p 9200:9200 -d elasticsearch -Des.path.repo=/var/backup/elasticsearch

3）我不明白这句话的作用：

docker run --link elasticsearch:elasticsearch -e ELASTICSEARCH_URL="elasticsearch:9200" -e SNAPSHOT_VOLUME="/var/backup/elasticsearch" -e S3_URL="your S3 url" -e S3_ACCESS_KEY="your S3 access key" -e S3_SECRET_KEY="your S3 secret key"

是否有更多接受/记录的方法将弹性搜索数据连续推送到外部存储桶？对于使用上述存储库需要采取的步骤的任何建议和/或照明将不胜感激。

Answer 1

参考文档

elasticsearch-backup-snapshot-and-restore-on-aws-s3

opendistro

Dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:7.6.2

#this is to tell that we are passing these arguments at runtime
ARG ENV_VAR_AWS_ACCESS_KEY_ID
ARG ENV_VAR_AWS_SECRET_ACCESS_KEY

ENV AWS_ACCESS_KEY_ID ${ENV_VAR_AWS_ACCESS_KEY_ID}
ENV AWS_SECRET_ACCESS_KEY ${ENV_VAR_AWS_SECRET_ACCESS_KEY}
ENV xpack.security.enabled 'false'
ENV  xpack.monitoring.enabled 'false'
ENV xpack.graph.enabled 'false'
ENV xpack.watcher.enabled 'false'
ENV discovery.type 'single-node'
ENV bootstrap.memory_lock 'true'
ENV indices.memory.index_buffer_size '30%'

RUN /usr/share/elasticsearch/bin/elasticsearch-plugin install --batch repository-s3
RUN /usr/share/elasticsearch/bin/elasticsearch-keystore create
RUN echo $AWS_ACCESS_KEY_ID | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.access_key
RUN echo $AWS_SECRET_ACCESS_KEY | /usr/share/elasticsearch/bin/elasticsearch-keystore add --stdin s3.client.default.secret_key

register.json

{
  "type": "s3",
  "settings": {
    "bucket": "elk-backup-codeaprendiz"
  }
}

restoresnapshot.json

{
  "indices": "kibana*,my-index*",
  "ignore_unavailable": true,
  "include_global_state": false,
  "include_aliases": false,
  "partial": false,
  "rename_pattern": "kibana(.+)",
  "rename_replacement": "restored-kibana$1",
  "index_settings": {
    "index.blocks.read_only": false
  },
  "ignore_index_settings": [
    "index.refresh_interval"
  ]
}

snapshotsetting.json

{
  "indices": "kibana*,my-index*,-my-index-2016",
  "ignore_unavailable": true,
  "include_global_state": false,
  "partial": false
}

构建映像并部署

建立新图像

docker build  \
--build-arg ENV_VAR_AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID> \
--build-arg ENV_VAR_AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ACCESS_KEY> \
--tag=codeaprendiz/elasticsearch .

检查显示的图像

$ docker images | grep "codeaprendiz/elasticsearch"
codeaprendiz/elasticsearch                      latest              f06a06d5fd8a        36 seconds ago      796MB

运行图像

docker run -p 9200:9200 -p 9600:9600  codeaprendiz/elasticsearch

注册

在S3上注册您的仓库

$ curl -X PUT -H "Content-Type: application/json" -d @register.json "http://localhost:9200/_snapshot/my-s3-repository"
{"acknowledged":true}

拍摄快照

拍摄快照

$ curl -X PUT -H "Content-Type: application/json" -d @snapshotsetting.json "http://localhost:9200/_snapshot/my-s3-repository/firstsnap?wait_for_completion=true"
{"snapshot":{"snapshot":"firstsnap","uuid":"VpRaTS-eRr6TLqIOi9Zw2w","version_id":7060299,"version":"7.6.2","indices":[],"include_global_state":false,"state":"SUCCESS","start_time":"2020-05-16T14:13:06.219Z","start_time_in_millis":1589638386219,"end_time":"2020-05-16T14:13:06.624Z","end_time_in_millis":1589638386624,"duration_in_millis":405,"failures":[],"shards":{"total":0,"failed":0,"successful":0}}}

对应的docker日志

$ docker logs -f friendly_fermi
{"type": "server", "timestamp": "2020-05-16T14:13:06,646Z", "level": "INFO", "component": "o.e.s.SnapshotsService", "cluster.name": "docker-cluster", "node.name": "676c35dac6af", "message": "snapshot [my-s3-repository:firstsnap/VpRaTS-eRr6TLqIOi9Zw2w] started", "cluster.uuid": "Crq-wvoIQmuzm920sZr8MA", "node.id": "Q_xnc6qyRxy-BbvRLQNwlg"  }
{"type": "server", "timestamp": "2020-05-16T14:13:08,852Z", "level": "INFO", "component": "o.e.s.SnapshotsService", "cluster.name": "docker-cluster", "node.name": "676c35dac6af", "message": "snapshot [my-s3-repository:firstsnap/VpRaTS-eRr6TLqIOi9Zw2w] completed with state [SUCCESS]", "cluster.uuid": "Crq-wvoIQmuzm920sZr8MA", "node.id": "Q_xnc6qyRxy-BbvRLQNwlg"  }

请求快照

$ curl -X GET  "http://localhost:9200/_snapshot/my-s3-repository/firstsnap"
{"snapshots":[{"snapshot":"firstsnap","uuid":"VpRaTS-eRr6TLqIOi9Zw2w","version_id":7060299,"version":"7.6.2","indices":[],"include_global_state":false,"state":"SUCCESS","start_time":"2020-05-16T14:13:06.219Z","start_time_in_millis":1589638386219,"end_time":"2020-05-16T14:13:06.624Z","end_time_in_millis":1589638386624,"duration_in_millis":405,"failures":[],"shards":{"total":0,"failed":0,"successful":0}}]}

要检查快照的状态

$ curl -X GET "http://localhost:9200/_snapshot/_status"                
{"snapshots":[]}

查看所有快照

$ curl -X GET "http://localhost:9200/_snapshot/_all"
{"my-s3-repository":{"type":"s3","settings":{"bucket":"elk-backup-codeaprendiz"}}}

查看存储库中的所有快照

$ curl -X GET "http://localhost:9200/_snapshot/my-s3-repository/_all"
{"snapshots":[{"snapshot":"firstsnap","uuid":"VwFvTv3nSKOD5K8J3EBE2A","version_id":7060299,"version":"7.6.2","indices":[],"include_global_state":false,"state":"SUCCESS","start_time":"2020-05-14T14:45:46.358Z","start_time_in_millis":1589467546358,"end_time":"2020-05-14T14:45:46.561Z","end_time_in_millis":1589467546561,"duration_in_millis":203,"failures":[],"shards":{"total":0,"failed":0,"successful":0}}]}

还原快照

还原快照

$ curl -X POST -H "Content-Type: application/json" -d @restoresnapshot.json "http://localhost:9200/_snapshot/my-s3-repository/firstsnap/_restore"
{"snapshot":{"snapshot":"firstsnap","indices":[],"shards":{"total":0,"failed":0,"successful":0}}}

Link to git - basic

Link to git - intermediate

将docker数据的elasticsearch连续推送到S3

1 个答案: