Question

我有一个很大的多线程Python项目，显然某个地方存在内存泄漏。 DoctorThread向我显示了这些（缩短的）结果：

Partition of a set of 418 objects. Total size = 96792 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0     43  10    22792  24     22792  24 guppy.etc.Glue.Interface
     1     66  16    18480  19     41272  43 dict of guppy.etc.Glue.Owner
     2     25   6    18344  19     59616  62 dict of guppy.etc.Glue.Share
     3      8   2     8384   9     68000  70 guppy.etc.Glue.Share
     4     86  21     6696   7     74696  77 dict (no owner)
     5     22   5     6160   6     80856  84 guppy.etc.Glue.Owner
     6     37   9     2608   3     83464  86 dict (no owner), dict of guppy.etc.Glue.Interface
     7     28   7     2464   3     85928  89 guppy.heapy.heapyc.HeapView
     8     11   3     1840   2     87768  91 <Nothing>
     9      2   0     1112   1     88880  92 __builtin__.cell
<24 more rows. Type e.g. '_.more' to view.>
Partition of a set of 23178 objects. Total size = 1604224 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  11135  48   801440  50    801440  50 list
     1  11153  48   602408  38   1403848  88 tuple
     [...]
<95 more rows. Type e.g. '_.more' to view.>
Partition of a set of 45140 objects. Total size = 2987568 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  22114  49  1591936  53   1591936  53 list
     1  22133  49  1195328  40   2787264  93 tuple
     [...]
<95 more rows. Type e.g. '_.more' to view.>
Partition of a set of 66115 objects. Total size = 4337720 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  32524  49  2341216  54   2341216  54 list
     1  32513  49  1755848  40   4097064  94 tuple
     [...]
<104 more rows. Type e.g. '_.more' to view.>
Partition of a set of 88355 objects. Total size = 5739128 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  43644  49  3141856  55   3141856  55 list
     1  43633  49  2356328  41   5498184  96 tuple
     [...]
<104 more rows. Type e.g. '_.more' to view.>
Partition of a set of 110380 objects. Total size = 7097992 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  54734  50  3940576  56   3940576  56 list
     1  54753  50  2956808  42   6897384  97 tuple
     [...]
<97 more rows. Type e.g. '_.more' to view.>

如您所见，list和tuple引荐来源网址的数量逐渐增加。而且它永远不会停止增长。这两个条目是唯一不断增加的条目。

DoctorThread类如下：

class DoctorThread(threading.Thread):
    def __init__(self):
        super(DoctorThread, self).__init__()
        self.daemon = True
        self.hp = guppy.hpy()

    def run(self):
        time.sleep(5)
        logging.info("Doctor Thread started - taking heap snapshots")
        before_heap = self.hp.heap()

        while not PippinNetwork.is_shutdown():
            gc.collect()
            leftover = self.hp.heap() - before_heap
            print(leftover.byrcs)
            time.sleep(2.0)

内存消耗同样在增加。我如何找到此泄漏的罪魁祸首？

更新：已解决。

事后看来，这是猖ramp的list.append((object, object))。孔雀鱼的结果本来可以这样解释。

Answer 1

万一大型源代码中发生内存泄漏，您可以在python中使用“ pympler”来跟踪将来的工作中的参考。

from pympler import muppy

 from pympler import summary

all_objects = muppy.get_objects()

sum1 = summary.summarize(all_objects)
 summary.print_(sum1)

您也可以过滤对象。仅包括那些您有疑问的对象，然后您可以查看哪个对象或变量导致内存泄漏，然后查看其背后的逻辑。

我建议使用这种方法，因为您不必缩减逻辑（因为您的项目将拥有庞大的代码）

如果需要澄清，您可以在评论中提问。

在一个非常大的项目中发现内存泄漏

更新：已解决。

1 个答案: