如何确保AWS S3 cli文件完全同步而不会丢失文件

时间:2018-09-10 15:27:04

标签: bash amazon-web-services amazon-s3 command-line-interface

我正在研究一个项目,该项目从S3上的MongoDB中获取数据库备份,并将其放到暂存盒中以供当天使用。我今天在一次手动运行中注意到,我得到了这个输出。通常,它显示每个文件的副本,但是今天我收到连接重置错误消息,或者* .15文件之一在操作完成后没有被复制。

这是我正在使用的AWS CLI命令:

aws s3 cp ${S3_PATH} ${BACKUP_PRODUCTION_PATH}/ --recursive

这是我回来的输出的摘录:

download: s3://myorg-mongo-backups-raw/production/daily/2018-09-10/080001/data/s-ds063192-a1/myorg-production/myorg-production.10
 to ../../data/db/myorg-production/myorg-production.10
download: s3://myorg-mongo-backups-raw/production/daily/2018-09-10/080001/data/s-ds063192-a1/myorg-production/myorg-production.11
 to ../../data/db/myorg-production/myorg-production.11
download: s3://myorg-mongo-backups-raw/production/daily/2018-09-10/080001/data/s-ds063192-a1/myorg-production/myorg-production.12
 to ../../data/db/myorg-production/myorg-production.12
download: s3://myorg-mongo-backups-raw/production/daily/2018-09-10/080001/data/s-ds063192-a1/myorg-production/myorg-production.13
 to ../../data/db/myorg-production/myorg-production.13
download s3://myorg-mongo-backups-raw/production/daily/2018-09-10/080001/data/s-ds063192-a1/myorg-production/myorg-production.14
to ../../data/db/myorg-production/myorg-production.14
download failed: s3://myorg-mongo-backups-raw/production/daily/2018-09-10/080001/data/s-ds063192-a1/myorg-production/myorg-produc
tion.15 to ../../data/db/myorg-production/myorg-production.15 ("Connection broken: error(104, 'Connection reset by peer')", error
(104, 'Connection reset by peer'))
download: s3://myorg-mongo-backups-raw/production/daily/2018-09-10/080001/data/s-ds063192-a1/myorg-production/myorg-production.16
to ../../data/db/myorg-production/myorg-production.16

如何确保将给定S3路径中的数据完全复制到目标路径,而没有任何连接问题,文件丢失等? AWS工具的sync命令是否是更好的选择?还是我应该尝试其他东西?

谢谢!

0 个答案:

没有答案