如何从Python释放内存回到操作系统?

时间:2014-07-03 23:40:59

标签: python python-2.7 garbage-collection multiprocessing psutil

我的代码与此类似:

def memoryIntensiveFunction(x):
    largeTempVariable = Intermediate(x)
    processFunction(largeTempVariable,x)

问题是变量temp在我的测试用例中类似于500 MB,但是当memoryIntensiveFunction完成时,该空间不会返回到操作系统。我知道这是因为使用guppy工具进行的内存分析表明largeTempVariable已被释放(即在Python中),但psutil表明它不是。我认为我看到了here所描述的效果。问题是这个过程是长时间运行的(即几小时),memoryIntensiveFunction在开始时运行而不再运行,因此我不得不携带500mb左右几个小时。

我找到的一个解决方案herehere建议使用单独的流程。多处理会产生自己的成本,但在我的情况下它是值得的。但是,这需要重构memoryIntensiveFunction个调用方以接收x作为返回值,而不是将其视为已修改。真正的杀手是我的对象x不可选(它大量使用boost python扩展)。要使x可选,需要做很多工作。

有没有我不考虑的选择?

1 个答案:

答案 0 :(得分:2)

这似乎很好奇,我试图重现你的问题,似乎简单的“del”很多。 要演示,您可以运行以下代码:

import itertools
import pdb

def test():
    a = "a"
    for _ in itertools.repeat(None, 30):
        a += a
    pdb.set_trace()
    del a
    pdb.set_trace()

test()

在第一个断点处你会看到它使用大约1GB的ram(你想要python3.3条目):

 Private  +   Shared  =  RAM used       Program

  4.0 KiB +   9.0 KiB =  13.0 KiB       VisualGDB-DisownTTY-r1
  4.0 KiB +  15.0 KiB =  19.0 KiB       sharing-tests
  4.0 KiB +  19.5 KiB =  23.5 KiB       dhcpcd
  4.0 KiB +  31.5 KiB =  35.5 KiB       gdb
  4.0 KiB +  36.0 KiB =  40.0 KiB       vim [deleted]
  4.0 KiB +  38.0 KiB =  42.0 KiB       systemd-udevd
 40.0 KiB +  10.0 KiB =  50.0 KiB       init
 24.0 KiB + 135.0 KiB = 159.0 KiB       agetty (6)
 12.0 KiB + 150.0 KiB = 162.0 KiB       su (3)
 88.0 KiB + 103.0 KiB = 191.0 KiB       syslog-ng (2)
152.0 KiB +  55.0 KiB = 207.0 KiB       crond
172.0 KiB +  81.0 KiB = 253.0 KiB       python3.4
580.0 KiB + 220.5 KiB = 800.5 KiB       sshd (3)
768.0 KiB + 932.0 KiB =   1.7 MiB       bash (13)
  2.8 MiB + 118.0 KiB =   2.9 MiB       mongod
  7.4 MiB + 109.0 KiB =   7.5 MiB       tmux [deleted] (2)
  1.0 GiB +   1.2 MiB =   1.0 GiB       python3.3
---------------------------------
                          1.0 GiB
=================================

然后在第二个断点处,在我们将变量释放后释放内存:

 Private  +   Shared  =  RAM used       Program

  4.0 KiB +   9.0 KiB =  13.0 KiB       VisualGDB-DisownTTY-r1
  4.0 KiB +  15.0 KiB =  19.0 KiB       sharing-tests
  4.0 KiB +  19.5 KiB =  23.5 KiB       dhcpcd
  4.0 KiB +  31.5 KiB =  35.5 KiB       gdb
  4.0 KiB +  36.0 KiB =  40.0 KiB       vim [deleted]
  4.0 KiB +  38.0 KiB =  42.0 KiB       systemd-udevd
 40.0 KiB +  10.0 KiB =  50.0 KiB       init
 24.0 KiB + 135.0 KiB = 159.0 KiB       agetty (6)
 12.0 KiB + 150.0 KiB = 162.0 KiB       su (3)
 88.0 KiB + 103.0 KiB = 191.0 KiB       syslog-ng (2)
152.0 KiB +  55.0 KiB = 207.0 KiB       crond
172.0 KiB +  81.0 KiB = 253.0 KiB       python3.4
584.0 KiB + 220.5 KiB = 804.5 KiB       sshd (3)
768.0 KiB + 928.0 KiB =   1.7 MiB       bash (13)
  2.8 MiB + 118.0 KiB =   2.9 MiB       mongod
  5.1 MiB +   1.2 MiB =   6.3 MiB       python3.3
  7.4 MiB + 109.0 KiB =   7.5 MiB       tmux [deleted] (2)
---------------------------------
                         20.3 MiB
=================================

现在,如果我们从函数中删除“del”,并在test()之后立即设置断点:

import itertools
import pdb

def test():
    a = "a"
    for _ in itertools.repeat(None, 30):
        a += a
    pdb.set_trace()

test()
pdb.set_trace()

在我们终止之前,内存确实不会被释放:

 Private  +   Shared  =  RAM used       Program

  4.0 KiB +   9.0 KiB =  13.0 KiB       VisualGDB-DisownTTY-r1
  4.0 KiB +  15.0 KiB =  19.0 KiB       sharing-tests
  4.0 KiB +  19.5 KiB =  23.5 KiB       dhcpcd
  4.0 KiB +  31.5 KiB =  35.5 KiB       gdb
  4.0 KiB +  36.0 KiB =  40.0 KiB       vim [deleted]
  4.0 KiB +  38.0 KiB =  42.0 KiB       systemd-udevd
 40.0 KiB +  10.0 KiB =  50.0 KiB       init
 24.0 KiB + 135.0 KiB = 159.0 KiB       agetty (6)
 12.0 KiB + 150.0 KiB = 162.0 KiB       su (3)
160.0 KiB +  53.0 KiB = 213.0 KiB       crond
172.0 KiB +  81.0 KiB = 253.0 KiB       python3.4
628.0 KiB + 219.5 KiB = 847.5 KiB       sshd (3)
836.0 KiB + 152.0 KiB = 988.0 KiB       syslog-ng (2)
752.0 KiB + 957.0 KiB =   1.7 MiB       bash (13)
  2.8 MiB + 113.0 KiB =   2.9 MiB       mongod
  7.4 MiB + 108.0 KiB =   7.6 MiB       tmux [deleted] (2)
  1.0 GiB +   1.1 MiB =   1.0 GiB       python3.3
---------------------------------
                          1.0 GiB
=================================

所以我的建议?只需在使用后删除它,并且不再需要它;)