如何在mongodb dabase实时之间传输大数据

时间:2017-11-16 03:27:05

标签: mongodb pymongo

现在我用python代码传输数据:

for k in length:
    db[collection].find().limit(step).skip(k * step) # step is 800 in my case

尝试从远程mongodb服务器传输数据的超过2个处理,但速度没有增加太多。消耗的CPU总是保持103-130%(通过扫视监视,2核心)。资源有限(小于100G磁盘)的机器同时生成大数据,因此我无法暂停并通过导出和导入传输数据。

日志显示如下。对于Mongodb的一个查询,成本超过1秒。那么有没有可能提高速度?

INFO 2017-11-16 03:22:48,345 4961 140669104162560  consumer_mongodb.py:96 9125
INFO 2017-11-16 03:22:48,345 4961 140669104162560  consumer_mongodb.py:97 process1
INFO 2017-11-16 03:22:48,985 4965 140669104162560  consumer_mongodb.py:96 3378
INFO 2017-11-16 03:22:48,985 4965 140669104162560  consumer_mongodb.py:97 process2
INFO 2017-11-16 03:22:51,919 4955 140669104162560  consumer_mongodb.py:137 sleeping
INFO 2017-11-16 03:22:54,984 4965 140669104162560  consumer_mongodb.py:96 3379
INFO 2017-11-16 03:22:54,985 4965 140669104162560  consumer_mongodb.py:97 process2
INFO 2017-11-16 03:22:56,926 4955 140669104162560  consumer_mongodb.py:137 sleeping
INFO 2017-11-16 03:23:01,932 4955 140669104162560  consumer_mongodb.py:137 sleeping
INFO 2017-11-16 03:23:02,361 4965 140669104162560  consumer_mongodb.py:96 3380
INFO 2017-11-16 03:23:02,363 4965 140669104162560  consumer_mongodb.py:97 process2
INFO 2017-11-16 03:23:02,643 4961 140669104162560  consumer_mongodb.py:96 9126
INFO 2017-11-16 03:23:02,644 4961 140669104162560  consumer_mongodb.py:97 process1
INFO 2017-11-16 03:23:06,938 4955 140669104162560  consumer_mongodb.py:137 sleeping
INFO 2017-11-16 03:23:09,479 4965 140669104162560  consumer_mongodb.py:96 3381
INFO 2017-11-16 03:23:09,480 4965 140669104162560  consumer_mongodb.py:97 process2
INFO 2017-11-16 03:23:11,944 4955 140669104162560  consumer_mongodb.py:137 sleeping

0 个答案:

没有答案