Question

有人可以向我解释一下mongo在同时更新单个文档时分配大量内存的情况。

我在说什么：

我有一个结构如下的文件：

{ id, data: [{id, status}] }

数据字段内部有很多对象（在我的测试案例中为5000个对象）。

在测试中，我只更新单个文档中的每个 data 对象。

当我同步执行时，没有额外的内存分配。但是当我并行更新文档时，mongo会分配超过2GB的额外内存。

在第一种情况下（同步）有分配：

{
    "pageheap_free_bytes" : 23552000,
    "pageheap_unmapped_bytes" : 0,
    "max_total_thread_cache_bytes" : NumberLong(1073741824),
    "current_total_thread_cache_bytes" : 994032,
    "total_free_bytes" : 7793496,
    "central_cache_free_bytes" : 4958056,
    "transfer_cache_free_bytes" : 1841408,
    "thread_cache_free_bytes" : 994032,
    "aggressive_memory_decommit" : 0,
    "pageheap_committed_bytes" : 112832512,
    "pageheap_scavenge_count" : 0,
    "pageheap_commit_count" : 85,
    "pageheap_total_commit_bytes" : 112832512,
    "pageheap_decommit_count" : 0,
    "pageheap_total_decommit_bytes" : 0,
    "pageheap_reserve_count" : 85,
    "pageheap_total_reserve_bytes" : 112832512,
    "spinlock_total_delay_ns" : NumberLong(1169495344),
    "formattedString" : "------------------------------------------------\nMALLOC:       81487592 (   77.7 MiB) Bytes in use by application\nMALLOC: +     23552000 (   22.5 MiB) Bytes in page heap freelist\nMALLOC: +      4958056 (    4.7 MiB) Bytes in central cache freelist\nMALLOC: +      1841408 (    1.8 MiB) Bytes in transfer cache freelist\nMALLOC: +       993456 (    0.9 MiB) Bytes in thread cache freelists\nMALLOC: +      2318592 (    2.2 MiB) Bytes in malloc metadata\nMALLOC:   ------------\nMALLOC: =    115151104 (  109.8 MiB) Actual memory used (physical + swap)\nMALLOC: +            0 (    0.0 MiB) Bytes released to OS (aka unmapped)\nMALLOC:   ------------\nMALLOC: =    115151104 (  109.8 MiB) Virtual address space used\nMALLOC:\nMALLOC:           2618              Spans in use\nMALLOC:             25              Thread heaps in use\nMALLOC:           4096              Tcmalloc page size\n------------------------------------------------\nCall ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).\nBytes released to the OS take up virtual address space but no physical memory.\n"
}

第二个：

{
    "pageheap_free_bytes" : 1020854272,
    "pageheap_unmapped_bytes" : 13770752,
    "max_total_thread_cache_bytes" : NumberLong(1073741824),
    "current_total_thread_cache_bytes" : 1996120,
    "total_free_bytes" : 24262888,
    "central_cache_free_bytes" : 19604560,
    "transfer_cache_free_bytes" : 2662208,
    "thread_cache_free_bytes" : 1996120,
    "aggressive_memory_decommit" : 0,
    "pageheap_committed_bytes" : NumberLong(2859118592),
    "pageheap_scavenge_count" : 51,
    "pageheap_commit_count" : 2781,
    "pageheap_total_commit_bytes" : NumberLong(2886373376),
    "pageheap_decommit_count" : 51,
    "pageheap_total_decommit_bytes" : 27254784,
    "pageheap_reserve_count" : 2690,
    "pageheap_total_reserve_bytes" : NumberLong(2872889344),
    "spinlock_total_delay_ns" : NumberLong(158962109950),
    "formattedString" : "------------------------------------------------\nMALLOC:     1814002008 ( 1730.0 MiB) Bytes in use by application\nMALLOC: +   1020854272 (  973.6 MiB) Bytes in page heap freelist\nMALLOC: +     19604560 (   18.7 MiB) Bytes in central cache freelist\nMALLOC: +      2662208 (    2.5 MiB) Bytes in transfer cache freelist\nMALLOC: +      1995544 (    1.9 MiB) Bytes in thread cache freelists\nMALLOC: +     12574976 (   12.0 MiB) Bytes in malloc metadata\nMALLOC:   ------------\nMALLOC: =   2871693568 ( 2738.7 MiB) Actual memory used (physical + swap)\nMALLOC: +     13770752 (   13.1 MiB) Bytes released to OS (aka unmapped)\nMALLOC:   ------------\nMALLOC: =   2885464320 ( 2751.8 MiB) Virtual address space used\nMALLOC:\nMALLOC:          19410              Spans in use\nMALLOC:             28              Thread heaps in use\nMALLOC:           4096              Tcmalloc page size\n------------------------------------------------\nCall ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).\nBytes released to the OS take up virtual address space but no physical memory.\n"
}

如您所见，应用程序正在使用的字节和页面堆空闲列表中的字节从77.7 / 22.5 MB增加到1730 / 973.6 MB。

因此，如果我多次运行并行代码，这些数字将线性增加。

有人可以解释mongo处理并发查询时发生了什么以及如何防止分配吗？

在测试中，我使用的是mongoDb 4.0.4（here是docker映像）和.net core client。

Answer 1

当您执行顺序更新时，您使用的是单个线程。仅使用单个连接。每个连接段最多需要1MB。 as per docs

如果您要进行并行更新，则它将具有多个连接段，每个连接段将使用1 MB，这将增加内存使用量。

并发更新单个文档时，MongoDB会分配大量内存

1 个答案: