我使用Java API将1亿个数据插入Mongodb(50%的列被索引,而不是由于业务逻辑而导致的批量插入)。
表和索引结构:
db.gm_std_measurements.findOne();
{
"_id" : ObjectId("530b6340e4b033fabd61fb99"),
"fkDataSeriesId" : 421,
"measDateUtc" : "2014-10-10 12:00:00",
"measDateSite" : "2014-03-15 12:00:00",
"project_id" : 379,
"measvalue" : 597.516583008608,
"refMeas" : false,
"reliability" : 1
}
db.gm_std_measurements.getIndexes();
[
{
"v" : 1,
"key" : {
"_id" : 1
},
"ns" : "testdb.gm_std_measurements",
"name" : "_id_"
},
{
"v" : 1,
"key" : {
"fkDataSeriesId" : 1,
"measDateUtc" : 1,
"measDateSite" : 1,
"project_id" : 1
},
"ns" : "testdb.gm_std_measurements",
"name" : "default_mongodb_test_index"
}
]
一开始,mongostat表示速度非常好,每秒约20-30k次插入。但是之后一段时间,性能下降非常快,系统负载为5-10。可能是什么原因?
正如所观察到的,很多时候,mongostat似乎被冻结(或mongod被冻结),因为根本没有插入,并且“锁定db”的跟踪数据也是0.0%,这是正常的?
非常感谢!
以下是mongostat的一些输出:
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time
39520 *0 *0 *0 0 1|0 0 160m 506m 67m 3 testdb:61.6% 0 0|0 0|1 9m 2k 4 15:58:26
36010 *0 *0 *0 0 1|0 0 160m 506m 83m 1 testdb:55.9% 0 0|0 0|0 8m 2k 4 15:58:27
33793 *0 *0 *0 0 1|0 0 288m 762m 92m 3 testdb:57.8% 0 0|0 0|0 7m 2k 4 15:58:28
32061 *0 *0 *0 0 1|0 0 288m 762m 113m 0 testdb:55.9% 0 0|0 0|0 7m 2k 4 15:58:29
32302 *0 *0 *0 0 1|0 0 288m 762m 110m 1 testdb:60.2% 0 0|0 0|1 7m 2k 4 15:58:30
31283 *0 *0 *0 0 1|0 0 288m 762m 138m 0 testdb:57.1% 0 0|0 0|1 7m 2k 4 15:58:31
1126 *0 *0 *0 0 1|0 0 544m 1.25g 367m 0 testdb:3.4% 0 0|0 0|1 258k 2k 4 15:58:55
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time
18330 *0 *0 *0 0 1|0 0 544m 1.25g 369m 1 testdb:40.8% 0 0|0 0|1 4m 2k 4 15:58:56
4235 *0 *0 *0 0 1|0 0 544m 1.25g 395m 0 testdb:7.3% 0 0|0 0|1 974k 2k 4 15:58:57
*0 *0 *0 *0 0 1|0 0 544m 1.25g 395m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:58:58
*0 *0 *0 *0 0 1|0 0 544m 1.25g 395m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:58:59
*0 *0 *0 *0 0 1|0 0 544m 1.25g 395m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:00
*0 *0 *0 *0 0 1|0 0 544m 1.25g 395m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:01
*0 *0 *0 *0 0 1|0 0 544m 1.25g 395m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:02
*0 *0 *0 *0 0 1|0 0 544m 1.25g 395m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:04
20083 *0 *0 *0 0 1|0 0 544m 1.25g 378m 0 .:23.4% 0 0|0 0|1 4m 2k 4 15:59:05
28595 *0 *0 *0 0 1|0 0 544m 1.25g 404m 0 testdb:60.0% 0 0|0 0|0 6m 2k 4 15:59:06
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time
26415 *0 *0 *0 0 1|0 0 544m 1.25g 381m 0 testdb:60.8% 0 0|0 0|1 6m 2k 4 15:59:07
27161 *0 *0 *0 0 1|0 0 544m 1.25g 411m 0 testdb:59.5% 0 0|0 0|1 6m 2k 4 15:59:08
25550 *0 *0 *0 0 1|0 0 544m 1.25g 397m 0 testdb:56.6% 0 0|0 0|1 5m 2k 4 15:59:09
26245 *0 *0 *0 0 1|0 0 544m 1.25g 429m 0 testdb:60.0% 0 0|0 0|1 6m 2k 4 15:59:10
27836 *0 *0 *0 0 1|0 0 544m 1.25g 444m 0 testdb:60.0% 0 0|0 0|1 6m 2k 4 15:59:11
27041 *0 *0 *0 0 1|0 0 544m 1.25g 422m 0 testdb:62.2% 0 0|0 0|1 6m 2k 4 15:59:12
26522 *0 *0 *0 0 1|0 0 544m 1.25g 463m 0 testdb:58.4% 0 0|0 0|1 6m 2k 4 15:59:13
27195 *0 *0 *0 0 1|0 0 544m 1.25g 475m 0 testdb:60.1% 0 0|0 0|1 6m 2k 4 15:59:14
25610 *0 *0 *0 0 1|0 0 1.03g 2.25g 500m 1 testdb:57.6% 0 0|0 0|1 5m 2k 4 15:59:15
25501 *0 *0 *0 0 1|0 0 1.03g 2.25g 474m 0 testdb:64.7% 0 0|0 0|1 5m 2k 4 15:59:16
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time
27446 *0 *0 *0 0 1|0 0 1.03g 2.25g 489m 0 testdb:58.2% 0 0|1 0|1 6m 2k 4 15:59:17
27113 *0 *0 *0 0 1|0 0 1.03g 2.25g 515m 1 testdb:57.2% 0 0|1 0|1 6m 2k 4 15:59:18
25383 *0 *0 *0 0 1|0 0 1.03g 2.25g 524m 0 testdb:59.9% 0 0|0 0|1 5m 2k 4 15:59:19
27506 *0 *0 *0 0 1|0 0 1.03g 2.25g 546m 1 testdb:61.3% 0 0|0 0|1 6m 2k 4 15:59:20
14901 2 *0 *0 0 1|0 0 1.03g 2.25g 498m 0 testdb:32.8% 0 0|1 0|1 3m 2k 4 15:59:21
9026 *0 *0 *0 0 1|0 0 1.03g 2.25g 501m 0 .:62.5% 0 0|1 0|1 2m 2k 4 15:59:24
16834 *0 *0 *0 0 1|0 1 1.03g 2.25g 506m 0 .:73.9% 0 0|1 0|1 3m 3k 4 15:59:25
25975 *0 *0 *0 0 1|0 0 1.03g 2.25g 521m 0 testdb:60.8% 0 0|0 0|1 5m 2k 4 15:59:26
23389 *0 *0 *0 0 1|0 0 1.03g 2.25g 525m 0 testdb:58.4% 0 0|0 0|1 5m 2k 4 15:59:27
27226 *0 *0 *0 0 1|0 0 1.03g 2.25g 584m 0 testdb:55.0% 0 0|0 0|1 6m 2k 4 15:59:28
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time
26362 *0 *0 *0 0 1|0 0 1.03g 2.25g 541m 0 testdb:56.3% 0 0|1 0|1 6m 2k 4 15:59:31
2658 *0 *0 *0 0 1|0 0 1.03g 2.25g 564m 0 .:64.2% 0 0|0 0|1 611k 3k 4 15:59:32
*0 *0 *0 *0 0 1|0 0 1.03g 2.25g 564m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:34
*0 *0 *0 *0 0 1|0 0 1.03g 2.25g 564m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:35
*0 *0 *0 *0 0 1|0 0 1.03g 2.25g 564m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:36
2777 *0 *0 *0 0 1|0 0 1.03g 2.25g 583m 0 testdb:4.8% 0 0|0 0|1 638k 2k 4 15:59:37
*0 *0 *0 *0 0 1|0 0 1.03g 2.25g 584m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:38
*0 *0 *0 *0 0 1|0 0 1.03g 2.25g 584m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:39
*0 *0 *0 *0 0 1|0 0 1.03g 2.25g 584m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:40
*0 *0 *0 *0 0 1|0 0 1.03g 2.25g 584m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:41
insert query update delete getmore command flushes mapped vsize res faults locked db idx miss % qr|qw ar|aw netIn netOut conn time
*0 *0 *0 *0 0 1|0 0 1.03g 2.25g 584m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 15:59:42
19823 *0 *0 *0 0 1|0 0 1.03g 2.25g 549m 0 testdb:57.8% 0 0|1 0|1 4m 2k 4 15:59:43
25267 *0 *0 *0 0 1|0 0 1.03g 2.25g 561m 0 testdb:60.4% 0 0|0 0|1 5m 2k 4 15:59:44
26489 *0 *0 *0 0 1|0 0 1.03g 2.25g 601m 0 testdb:58.8% 0 0|0 0|1 6m 2k 4 15:59:45
26516 *0 *0 *0 0 1|0 0 1.03g 2.25g 604m 0 testdb:58.4% 0 0|0 0|1 1m 2k 4 16:00:26
*0 *0 *0 *0 0 1|0 0 2.03g 4.25g 868m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 16:00:27
*0 *0 *0 *0 0 1|0 0 2.03g 4.25g 868m 0 testdb:0.0% 0 0|0 0|1 62b 2k 4 16:00:33
2775 *0 *0 *0 0 1|0 0 2.03g 4.25g 845m 0 testdb:0.8% 0 0|1 0|1 638k 3k 4 16:00:34
3886 *0 *0 *0 0 1|0 0 2.03g 4.25g 879m 0 .:30.5% 0 0|0 0|1 893k 2k 4 16:00:35
答案 0 :(得分:1)
您可以尝试删除索引,然后执行插入,在插入完成后,您可以创建索引。我认为这将是一个整体更快的情况。
您还可以在后台重新创建索引
db.collection.ensureIndex( { a: 1 }, { background: true } )
如果你想继续查询,但这会使索引创建更慢