有一个名为poor man's profiler的非常有用的gdb“脚本”。它调用此命令:
gdb -ex "set pagination 0" -ex "thread apply all bt" --batch -p $pid
适用于大多数Linux进程,但不适用于nginx Web服务器。
正常输出:
# gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p 5286
Using host libthread_db library "/lib/libthread_db.so.1".
[Thread debugging using libthread_db enabled]
[New Thread 0xb7d996c0 (LWP 5286)]
[New Thread 0xb588ab90 (LWP 5292)]
[New Thread 0xb608bb90 (LWP 5291)]
[New Thread 0xb688cb90 (LWP 5290)]
[New Thread 0xb708db90 (LWP 5289)]
[New Thread 0xb788eb90 (LWP 5288)]
0xffffe410 in __kernel_vsyscall ()
Thread 6 (Thread 0xb788eb90 (LWP 5288)):
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7e5d7a6 in epoll_wait () from /lib/libc.so.6
#2 0xb7ef4f3b in epoll_dispatch () from /usr/lib/libevent-1.3b.so.1
#3 0xb7ee963a in event_base_loop () from /usr/lib/libevent-1.3b.so.1
#4 0x08055537 in worker_libevent (arg=0x805f3a0) at thread.c:245
#5 0xb7ed2192 in start_thread () from /lib/libpthread.so.0
#6 0xb7e5d02e in clone () from /lib/libc.so.6
[cut]
Thread 1 (Thread 0xb7d996c0 (LWP 5286)):
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7e5d7a6 in epoll_wait () from /lib/libc.so.6
#2 0xb7ef4f3b in epoll_dispatch () from /usr/lib/libevent-1.3b.so.1
#3 0xb7ee963a in event_base_loop () from /usr/lib/libevent-1.3b.so.1
#4 0x0804f439 in main (argc=1, argv=0xbfbaff14) at memcached.c:4681
#0 0xffffe410 in __kernel_vsyscall ()
Nginx输出:
# gdb -ex "set pagination 0" -ex "thread apply all bt" -batch -p 6120
Using host libthread_db library "/lib/libthread_db.so.1".
0xffffe410 in __kernel_vsyscall ()
但如果使用“bt”代替“thread apply all bt”,它的效果很好:
# gdb -ex "set pagination 0" -ex "bt" -batch -p 6120
Using host libthread_db library "/lib/libthread_db.so.1".
0xffffe410 in __kernel_vsyscall ()
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7c83778 in epoll_wait () from /lib/libc.so.6
#2 0x080664c0 in ngx_epoll_process_events (cycle=0x80d14b8, timer=500, flags=1) at src/event/modules/ngx_epoll_module.c:530
#3 0x0805f73b in ngx_process_events_and_timers (cycle=0x80d14b8) at src/event/ngx_event.c:245
#4 0x080652b3 in ngx_worker_process_cycle (cycle=0x80d14b8, data=0x0) at src/os/unix/ngx_process_cycle.c:795
#5 0x08063ba1 in ngx_spawn_process (cycle=0x80d14b8, proc=0x80651fb <ngx_worker_process_cycle>, data=0x0, name=0x80a881d "worker process", respawn=-3) at src/os/unix/ngx_process.c:196
#6 0x080648d2 in ngx_start_worker_processes (cycle=0x80d14b8, n=8, type=-3) at src/os/unix/ngx_process_cycle.c:355
#7 0x0806581f in ngx_master_process_cycle (cycle=0xfffffffc) at src/os/unix/ngx_process_cycle.c:136
#8 0x0804d076 in main (argc=1, argv=0xbfd86544) at src/core/nginx.c:396
为什么?据我所知,“线程应用所有bt”必须工作,即使进程中只有一个执行线程。
更新
手动gdb连接并发出“info threads”命令。
# gdb -p 17461
GNU gdb 6.4
Copyright 2005 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i586-suse-linux".
Attaching to process 17461
Reading symbols from /export/depo/apache/linux/nginx-0.8.34/sbin/nginx...done.
Using host libthread_db library "/lib/libthread_db.so.1".
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /export/depo/mysql/linux/mysql-proxy-0.7.2/lib/libpcre.so.0...done.
Loaded symbols for /opt/gnu/mysql-proxy/lib/libpcre.so.0
Reading symbols from /usr/lib/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libz.so.1...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
0xffffe410 in __kernel_vsyscall ()
(gdb) info threads
(gdb) thread 0
Thread ID 0 not known.
(gdb) bt
#0 0xffffe410 in __kernel_vsyscall ()
#1 0xb7d0dd98 in __epoll_wait_nocancel () from /lib/libc.so.6
#2 0x08066e7d in ngx_epoll_process_events (cycle=0x816b708, timer=500, flags=1) at src/event/modules/ngx_epoll_module.c:530
#3 0x0805f4a2 in ngx_process_events_and_timers (cycle=0x816b708) at src/event/ngx_event.c:245
#4 0x08064d83 in ngx_worker_process_cycle (cycle=0x816b708, data=0x0) at src/os/unix/ngx_process_cycle.c:795
#5 0x08063661 in ngx_spawn_process (cycle=0x816b708, proc=0x8064ccb <ngx_worker_process_cycle>, data=0x0, name=0x80a42ed "worker process", respawn=-4) at src/os/unix/ngx_process.c:196
#6 0x080643a1 in ngx_start_worker_processes (cycle=0x816b708, n=8, type=-4) at src/os/unix/ngx_process_cycle.c:355
#7 0x0806595c in ngx_master_process_cycle (cycle=0x816b708) at src/os/unix/ngx_process_cycle.c:249
#8 0x0804cf7f in main (argc=1, argv=0xbfdb1874) at src/core/nginx.c:396
答案 0 :(得分:1)
thread apply all bt
为每个帖子做(惊喜!)bt
。它会在bt
导致错误的任何线程中立即停止(但不会发生这种情况)。
因此,如果您手动附加到nginx
,然后执行info threads
,然后继续重复thread N
和bt
,您可能会找到GDB的主题不能产生回溯。
确切知道GDB为该线程打印的内容可能很有趣。
您可能还想尝试使用当前版本的GDB(7.2)
答案 1 :(得分:0)
nginx不是多线程的(请参阅OpenSSL Locks上的此帖子)。这可以解释为什么你没有像你期望的那样看到线程数据的丰富。
nginx有一个主进程,而forks
(没有exec
)来创建worker。请参阅第90行周围的ngx_process.c
和ngx_spawn_process
。
我不知道PMP(穷人的探究者)对它有多好,但我相信你需要在set follow-fork-mode child
中gdb
。例如,请参阅gdbserver follow child。