为什么mongorestore痛苦地慢?

时间:2013-08-14 13:09:43

标签: mongodb ubuntu mongorestore

我收集了一个只有1个集合和两个索引的数据库。该系列大约有650万份文件。当我试图恢复它时,令我惊讶的是,这个过程非常缓慢。 一些统计数据如下:

Wed Aug 14 12:34:08.346         Progress: 333818/1378309050 0%  (bytes)
Wed Aug 14 12:34:11.077         Progress: 1530246/1378309050    0%  (bytes)
Wed Aug 14 12:34:26.177         Progress: 2714544/1378309050    0%  (bytes)
Wed Aug 14 12:34:30.145         Progress: 3355627/1378309050    0%  (bytes)
Wed Aug 14 12:34:34.504         Progress: 3895668/1378309050    0%  (bytes)
Wed Aug 14 12:34:53.246         Progress: 4334159/1378309050    0%  (bytes)
Wed Aug 14 12:34:56.318         Progress: 4963878/1378309050    0%  (bytes)
Wed Aug 14 12:34:59.545         Progress: 5617794/1378309050    0%  (bytes)
Wed Aug 14 12:35:08.042         Progress: 6923804/1378309050    0%  (bytes)
Wed Aug 14 12:35:16.424         Progress: 7342576/1378309050    0%  (bytes)
Wed Aug 14 12:35:23.168         Progress: 7987560/1378309050    0%  (bytes)
Wed Aug 14 12:35:29.703         Progress: 9295140/1378309050    0%  (bytes)
Wed Aug 14 12:35:38.582         Progress: 9943758/1378309050    0%  (bytes)
Wed Aug 14 12:35:43.574         Progress: 11128693/1378309050   0%  (bytes)
Wed Aug 14 12:35:46.008         Progress: 11982044/1378309050   0%  (bytes)
Wed Aug 14 12:35:50.134         Progress: 12421241/1378309050   0%  (bytes)
Wed Aug 14 12:35:54.548         Progress: 13166696/1378309050   0%  (bytes)
Wed Aug 14 12:35:58.152         Progress: 13837935/1378309050   1%  (bytes)

从上面的数据可以推断,总转储(在bson中)大约是1.3 Gigs。它正在让mongorestore令人失望的110秒恢复1%,即13 MB。

如果有人对此有解释,请告诉我。我希望我无意中做错了,因为这些数字与本世纪的计算标准相比太慢了。

修改


我再次使用以下两个选项运行命令,希望它们能加快进程:

--noobjcheck --noIndexRestore

但令我惊讶的是,这个过程现在更加缓慢!以下是一些统计数据。

Wed Aug 14 13:13:53.750     going into namespace [temp_raw_tweet_db.tweets]
Wed Aug 14 13:14:00.258         Progress: 871186/1378309050 0%  (bytes)
Wed Aug 14 13:14:04.424         Progress: 2070390/1378309050    0%  (bytes)
Wed Aug 14 13:14:07.482         Progress: 2921304/1378309050    0%  (bytes)
Wed Aug 14 13:14:11.895         Progress: 3647526/1378309050    0%  (bytes)
Wed Aug 14 13:14:57.028         Progress: 4984815/1378309050    0%  (bytes)
Wed Aug 14 13:15:01.015         Progress: 6202286/1378309050    0%  (bytes)
Wed Aug 14 13:15:05.051         Progress: 6797800/1378309050    0%  (bytes)
Wed Aug 14 13:15:08.402         Progress: 8133842/1378309050    0%  (bytes)
Wed Aug 14 13:15:12.712         Progress: 8872607/1378309050    0%  (bytes)
Wed Aug 14 13:15:15.259         Progress: 9964997/1378309050    0%  (bytes)
Wed Aug 14 13:15:19.266         Progress: 14684145/1378309050   1%  (bytes)
Wed Aug 14 13:15:22.364         Progress: 16154567/1378309050   1%  (bytes)
Wed Aug 14 13:15:29.627         Progress: 16754495/1378309050   1%  (bytes)
Wed Aug 14 13:15:35.225         Progress: 17726291/1378309050   1%  (bytes)
Wed Aug 14 13:15:39.447         Progress: 18333902/1378309050   1%  (bytes)
Wed Aug 14 13:15:43.717         Progress: 19055308/1378309050   1%  (bytes)
Wed Aug 14 13:15:46.481         Progress: 19305912/1378309050   1%  (bytes)
Wed Aug 14 13:15:49.902         Progress: 20038391/1378309050   1%  (bytes)
Wed Aug 14 13:15:53.868         Progress: 20389108/1378309050   1%  (bytes)
Wed Aug 14 13:15:58.578         Progress: 21127296/1378309050   1%  (bytes)
Wed Aug 14 13:16:03.706         Progress: 21837923/1378309050   1%  (bytes)
Wed Aug 14 13:16:56.512         Progress: 22092536/1378309050   1%  (bytes)
Wed Aug 14 13:16:59.035         Progress: 22583057/1378309050   1%  (bytes)
Wed Aug 14 13:17:02.313         Progress: 22817464/1378309050   1%  (bytes)
Wed Aug 14 13:17:05.044         Progress: 23178521/1378309050   1%  (bytes)
Wed Aug 14 13:17:26.023         Progress: 23434010/1378309050   1%  (bytes)
Wed Aug 14 13:17:39.161         Progress: 23670222/1378309050   1%  (bytes)
Wed Aug 14 13:17:42.846         Progress: 24049639/1378309050   1%  (bytes)
Wed Aug 14 13:17:59.125         Progress: 24284177/1378309050   1%  (bytes)
Wed Aug 14 13:18:02.722         Progress: 24515270/1378309050   1%  (bytes)
Wed Aug 14 13:18:06.827         Progress: 25018013/1378309050   1%  (bytes)
Wed Aug 14 13:18:09.234         Progress: 25253850/1378309050   1%  (bytes)
Wed Aug 14 13:18:14.282         Progress: 25617812/1378309050   1%  (bytes)
Wed Aug 14 13:18:46.296         Progress: 25983107/1378309050   1%  (bytes)
Wed Aug 14 13:18:51.303         Progress: 26604320/1378309050   1%  (bytes)
Wed Aug 14 13:18:55.500         Progress: 26971559/1378309050   1%  (bytes)
Wed Aug 14 13:19:00.656         Progress: 27444735/1378309050   1%  (bytes)
Wed Aug 14 13:19:04.100         Progress: 28064675/1378309050   2%  (bytes)

大约 4分钟从1%到2%。当然,这里存在严重的错误。

2 个答案:

答案 0 :(得分:9)

这是一个非常古老的主题,但最近我遇到了类似的问题,可能是出于不同的原因,并且遇到了这个问题。

如果您正在AWS上运行mongo,请务必使用正确的实例和卷类型。

T型实例具有CPU积分,将在运行大型mongorestore进程时耗尽。你的恢复将快速启动,然后慢下来爬行......它永远不会停止,但它需要数天才能完成。

如果你试图使用磁性EBS音量节省一些钱,这也是个坏主意。 SC1特别错误,因为它每秒的操作具有相似的信用额度...... mongorestore将烧掉你所有的IOP积分,无论你有多少分钟,之后速度将减少到每秒3次操作并且恢复可以采取周末完成。

我希望能帮助别人。

答案 1 :(得分:5)

不幸的是,这些数字并不罕见。 mongorestore收集了3亿美元。

你基本上有两种选择。

首先,只需进行长时间运行的恢复并在一夜之间运行:

nohup mongorestore [args] &

其次,复制数据库目录中的文件(默认为/data/db),而不是使用mongodump / mongorestore。如果您可以关闭数据库一段时间,这是最好的。否则,您必须使用文件系统快照或类似的。有关Mongo官方模糊的更多信息,请参阅此内容:http://docs.mongodb.org/manual/core/backups/