为什么Devel :: LeakTrace会泄漏内存？

时间：2019-07-12 11:08:58

标签： perl

我正在尝试学习有关如何检测Perl中的内存泄漏的更多信息。 我有这个程序：

p.pl ：

#! /usr/bin/env perl

use Devel::LeakTrace;
my $foo;
$foo = \$foo;

输出：

leaked SV(0xac2df8e0) from ./p.pl line 5
leaked SV(0xac2df288) from ./p.pl line 5

为什么这会泄漏两个标量（而不仅仅是一个）？

然后我通过valgrind运行它。首先，我创建了perl的调试版本：

$ perlbrew install perl-5.30.0 --as=5.30.0-D3L -DDEBUGGING \
  -Doptimize=-g3 -Accflags="-DDEBUG_LEAKING_SCALARS"
$ perlbrew use 5.30.0-D3L
$ cpanm Devel::LeakTrace

然后我按照perlhacktips中的建议运行valgrind设置PERL_DESTRUCT_LEVEL=2：

$  PERL_DESTRUCT_LEVEL=2 valgrind --leak-check=yes perl p.pl
==12479== Memcheck, a memory error detector
==12479== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12479== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==12479== Command: perl p.pl
==12479== 
leaked SV(0x4c27320) from p.pl line 5
leaked SV(0x4c26cc8) from p.pl line 5
==12479== 
==12479== HEAP SUMMARY:
==12479==     in use at exit: 105,396 bytes in 26 blocks
==12479==   total heap usage: 14,005 allocs, 13,979 frees, 3,011,508 bytes allocated
==12479== 
==12479== 16 bytes in 1 blocks are definitely lost in loss record 5 of 21
==12479==    at 0x483874F: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==12479==    by 0x484851A: note_changes (LeakTrace.xs:80)
==12479==    by 0x48488E3: XS_Devel__LeakTrace_hook_runops (LeakTrace.xs:126)
==12479==    by 0x32F0A2: Perl_pp_entersub (pp_hot.c:5237)
==12479==    by 0x2C0C50: Perl_runops_debug (dump.c:2537)
==12479==    by 0x1A2FD9: Perl_call_sv (perl.c:3043)
==12479==    by 0x1ACEE3: Perl_call_list (perl.c:5084)
==12479==    by 0x181233: S_process_special_blocks (op.c:10471)
==12479==    by 0x180989: Perl_newATTRSUB_x (op.c:10397)
==12479==    by 0x220D6C: Perl_yyparse (perly.y:295)
==12479==    by 0x3EE46B: S_doeval_compile (pp_ctl.c:3502)
==12479==    by 0x3F4F87: S_require_file (pp_ctl.c:4322)
==12479== 
==12479== LEAK SUMMARY:
==12479==    definitely lost: 16 bytes in 1 blocks
==12479==    indirectly lost: 0 bytes in 0 blocks
==12479==      possibly lost: 0 bytes in 0 blocks
==12479==    still reachable: 105,380 bytes in 25 blocks
==12479==         suppressed: 0 bytes in 0 blocks
==12479== Reachable blocks (those to which a pointer was found) are not shown.
==12479== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==12479== 
==12479== For counts of detected and suppressed errors, rerun with: -v
==12479== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

因此丢失了16个字节。但是，如果我注释掉use Devel::LeakTrace中的行p.pl并再次运行valgrind，则输出为：

==12880== Memcheck, a memory error detector
==12880== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==12880== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==12880== Command: perl p.pl
==12880== 
==12880== 
==12880== HEAP SUMMARY:
==12880==     in use at exit: 0 bytes in 0 blocks
==12880==   total heap usage: 1,770 allocs, 1,770 frees, 244,188 bytes allocated
==12880== 
==12880== All heap blocks were freed -- no leaks are possible
==12880== 
==12880== For counts of detected and suppressed errors, rerun with: -v
==12880== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

问题是：为什么Devel::LeakTrace导致内存泄漏？

2 个答案:

答案 0 :(得分：2)

似乎泄漏的内存甚至超过报告的valgrind。 每次创建新的SV时，Devel::LeakTrace都会在16 bytes structure called when中记录当前文件名和行号：

typedef struct {
    char *file;
    int line;
} when;

这些块与malloc()一起分配在line #80处，但似乎永远不会释放这些块。因此，创建的标量越多，泄漏的内存就越多。

一些背景信息

模块尝试从END{} phaser确定泄漏的SV。此时，所有已分配的SV应该已经超出了主程序的范围，并具有reference count decreased to zero，应该将其销毁。但是，如果由于某种原因引用计数未减少为零，则标量将不会被破坏和释放    从perl的内部内存管理池中。在这种情况下，标量被认为是模块泄漏的。

请注意，这与从操作中看到的内存泄漏不同 由例如malloc（）。当perl退出时，它仍然会 将所有泄漏的标量（从其内部内存池中）释放回系统内存池。

这意味着该模块不是要检测泄漏的系统内存。为此，我们可以使用valgrind。

该模块挂接到perl runops loop中，对于类型为OP_NEXTSTATE的每个OP，它将扫描所有arenas和其中的所有SV，以查找新的SV（即：从上一个OP_NEXTSTATE开始引入。

对于这个示例程序p.pl，我计算了31个竞技场，每个竞技场都包含71个SV的空间。这些SV几乎全部在运行时使用（其中大约2150个）。模块将每个SV保留在哈希值used中，其密钥等于SV的地址，其值等于分配了标量的when块（请参见上文）。然后，对于每个OP_NEXTSTATE，它可以扫描所有SV，并检查used哈希中是否不存在某些SV。

used哈希不是perl哈希（我想这是为了避免与 （模块尝试跟踪的已分配SV），而模块使用GLib hash tables。

补丁

为了跟踪分配的when块，我使用了一个名为when_hash的新glib哈希。然后，在模块打印泄漏的标量之后，可以通过在when中查找所有键来释放when_hash块。

我还发现该模块没有释放used哈希。据我所知，它应该调用glib g_hash_table_destroy()从END{}块中释放它。这是补丁：

LeakTrace.xs （已修补）：

#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"

#include <glib.h>


typedef struct {
    char *file;
    int line;
} when;

/* a few globals, never mind the mess for now */
GHashTable *used = NULL;
GHashTable *new_used = NULL;

/* cargo from Devel::Leak - wander the arena, see what SVs live */
typedef long used_proc _((void *,SV *,long));

/* PATCH: fix memory leaks */
/***************************/

GHashTable *when_hash = NULL;  /* store the allocated when blocks here */
static int have_run_end_hook = 0;  /* indicator to runops that we are done */
static runops_proc_t save_orig_run_ops; /* original runops function */

/* Called from END{}, i.e. from show_used() after having printed the leaks.
 * Free memory allocated for the when blocks */
static
void
free_when_block(gpointer key, gpointer value, gpointer user_data) {
    free(key);
}

static
void
do_cleanup() {
    /* this line was missing from the original show_used() */
    if (used) g_hash_table_destroy( used );

    if (when_hash) g_hash_table_foreach( when_hash, free_when_block, NULL );
    g_hash_table_destroy( when_hash );
    PL_runops = save_orig_run_ops;
    have_run_end_hook = 1;
}



/* END PATCH: fix memory leaks */
/*******************************/


static
long int
sv_apply_to_used(void *p, used_proc *proc, long n) {
    SV *sva;
    for (sva = PL_sv_arenaroot; sva; sva = (SV *) SvANY(sva)) {
        SV *sv = sva + 1;
        SV *svend = &sva[SvREFCNT(sva)];

        while (sv < svend) {
            if (SvTYPE(sv) != SVTYPEMASK) {
                n = (*proc) (p, sv, n);
            }
            ++sv;
        }
    }
    return n;
}
/* end Devel::Leak cargo */


static
long
note_used(void *p, SV* sv, long n) {
    when *old = NULL;

    if (used && (old = g_hash_table_lookup( used, sv ))) {
        g_hash_table_insert(new_used, sv, old);
        return n;
    }
    g_hash_table_insert(new_used, sv, p);
    return 1;
}

static
void
print_me(gpointer key, gpointer value, gpointer user_data) {
    when *w = value;
    char *type;

    switch SvTYPE((SV*)key) {
    case SVt_PVAV: type = "AV"; break;
    case SVt_PVHV: type = "HV"; break;
    case SVt_PVCV: type = "CV"; break;
    case SVt_RV:   type = "RV"; break;
    case SVt_PVGV: type = "GV"; break;
    default: type = "SV";
    }

    if (w->file) {
        fprintf(stderr, "leaked %s(0x%x) from %s line %d\n", 
        type, key, w->file, w->line);
    }
}

static
int
note_changes( char *file, int line ) {
    static when *w = NULL;
    int ret;

    /* PATCH */ 

    if (have_run_end_hook) return 0; /* do not enter after clean up is complete */
    /* if (!w) w = malloc(sizeof(when)); */
    if (!w) {
        w = malloc(sizeof(when));
        if (!when_hash) {
            /* store pointer to allocated blocks here */
            when_hash = g_hash_table_new( NULL, NULL );
        }
        g_hash_table_insert(when_hash, w, NULL); /* store address to w */
    }
    /* END PATCH */
    w->line = line;
    w->file = file;
    new_used = g_hash_table_new( NULL, NULL );
    if (sv_apply_to_used( w, note_used, 0 )) w = NULL;
    if (used) g_hash_table_destroy( used );
    used = new_used;
    return ret;
}

/* Now this bit of cargo is a derived from Devel::Caller */

static
int
runops_leakcheck(pTHX) {
    char *lastfile = 0;
    int lastline = 0;
    IV last_count = 0;

    while ((PL_op = CALL_FPTR(PL_op->op_ppaddr)(aTHX))) {
        PERL_ASYNC_CHECK();

        if (PL_op->op_type == OP_NEXTSTATE) {
            if (PL_sv_count != last_count) {
                note_changes( lastfile, lastline );
                last_count = PL_sv_count;
            }
            lastfile = CopFILE(cCOP);
            lastline = CopLINE(cCOP);
        }
    }

    note_changes( lastfile, lastline );

    TAINT_NOT;
    return 0;
}

MODULE = Devel::LeakTrace PACKAGE = Devel::LeakTrace

PROTOTYPES: ENABLE

void
hook_runops()
  PPCODE:
{
    note_changes(NULL, 0);
    PL_runops = runops_leakcheck;
}

void
reset_counters()
  PPCODE:
{
    if (used) g_hash_table_destroy( used );
    used = NULL;
    note_changes(NULL, 0);
}

void
show_used()
CODE:
{
    if (used) g_hash_table_foreach( used, print_me, NULL );
    /* PATCH */
    do_cleanup();  /* released allocated memory, restore original runops */
    /* END PATCH */
}

测试补丁

$ wget https://www.cpan.org/modules/by-module/Devel/Devel-LeakTrace-0.06.tar.gz
$ tar zxvf Devel-LeakTrace-0.06.tar.gz
$ cd Devel-LeakTrace-0.06
$ perlbrew use 5.30.0-D3L
# replace lib/Devel/LeakTrace.xs with my patch
$ perl Makefile.PL
$ make
$ make install  # <- installs the patch
# cd to test folder, then
$ PERL_DESTRUCT_LEVEL=2 valgrind --leak-check=yes perl p.pl
==25019== Memcheck, a memory error detector
==25019== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==25019== Using Valgrind-3.14.0 and LibVEX; rerun with -h for copyright info
==25019== Command: perl p.pl
==25019== 
leaked SV(0x4c26cd8) from p.pl line 5
leaked SV(0x4c27330) from p.pl line 5
==25019== 
==25019== HEAP SUMMARY:
==25019==     in use at exit: 23,324 bytes in 18 blocks
==25019==   total heap usage: 13,968 allocs, 13,950 frees, 2,847,004 bytes allocated
==25019== 
==25019== LEAK SUMMARY:
==25019==    definitely lost: 0 bytes in 0 blocks
==25019==    indirectly lost: 0 bytes in 0 blocks
==25019==      possibly lost: 0 bytes in 0 blocks
==25019==    still reachable: 23,324 bytes in 18 blocks
==25019==         suppressed: 0 bytes in 0 blocks
==25019== Reachable blocks (those to which a pointer was found) are not shown.
==25019== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==25019== 
==25019== For counts of detected and suppressed errors, rerun with: -v
==25019== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

答案 1 :(得分：1)

首先，valgrind在一个脚本中报告16个字节的泄漏内存，该脚本最多包含use Devel::LeakTrace。可能的泄漏与第四和第五行无关。在您的链接中，

NOTE 3: There are known memory leaks when there are compile-time errors 
within eval or require, seeing S_doeval in the call stack is a good sign 
of these. Fixing these leaks is non-trivial, unfortunately, but they must be fixed 
eventually.

由于我看到了行by 0x3F18E5: S_doeval_compile (pp_ctl.c:3502)，并且在您的示例中也看到了类似的行，因此我想说，这就是Devel::LeakTrace导致明显的内存泄漏的原因。 其次，对于原始脚本，Devel::LeakTrace仅报告（至少）第五行的循环引用引起的泄漏。您可以使用weaken中的Scalar::Util来查看此内容：

#! /usr/bin/env perl

use Devel::LeakTrace;
use Scalar::Util;
my $foo;
$foo = \$foo;
Scalar::Util::weaken($foo);

然后，perl p.pl将不会报告任何泄漏。我的猜测是，第一个脚本报告了两次泄漏，因为除了创建循环引用之外，perl还在$foo = \$foo处丢失了一个指针。当您削弱$foo显然可以解决两个问题时，会发生一些我无法理解的魔术。您可以通过调整原始脚本来查看此内容：

#! /usr/bin/env perl

use Devel::LeakTrace;
my $foo;
my $bar = \$foo;
$foo = $bar;

结果$foo应该相同，我们刚刚创建了$bar来保存引用。但是，在这种情况下，脚本仅报告一次泄漏。 因此，总而言之，我想说：1）Devel::LeakTrace有一个错误，独立于代码，显示为valgrind中的内存泄漏； 2）perl在原始脚本中创建一个循环引用并丢失了一个指针，这就是Devel::LeakTrace报告两次泄漏的原因。

相关问题
最新问题