如何使用Kernel Panic日志识别内存损坏的原因?

时间:2018-06-04 10:40:04

标签: c linux linux-kernel kernel addr2line

我的自定义硬件基于imax6q处理器。内核我正在使用 - Linux-boundary 4.1.15。我在自定义硬件设计上运行linux应用程序时遇到内核恐慌。那么我该如何调试它来澄清这是硬件相关还是软件问题?或者如何解决哪个内存正在破坏?

目前,我使用 addr2line -e vmlinux_with_debug_info 804a47e8 来解码内核恐慌消息。但它只在恐慌发生时显示内核源(程序计数器停止的地方)。

1)作为解码[804a47e8]时来自以下内核日志的示例,它仅提供内核源代码。所以你知道如何识别哪个内存会被破坏吗?或者我们可以获得更多细节吗?

前:

使用addr2line>>

root@osboxes:/home/debug# addr2line -e vmlinux 804a47e8
/yocto/fsl-community-bsp/build/workspace/sources/linux- 
boundary/lib/rbtree.c:457

因此源是rbtree.c而第457行将崩溃。

if (node->rb_right) {
node = node->rb_right; 
while (node->rb_left)    **//Line 457**
node=node->rb_left;
return (struct rb_node *)node;
}

在文件entry-armv.S第1219行指向addr2line

vectors_start:
>>W(b)    vector_rst    **//Line 1219**
W(b)    vector_und
W(ldr)    pc, __vectors_start + 0x1000
W(b)    vector_pabt
W(b)    vector_dabt
W(b)    vector_addrexcptn
W(b)    vector_irq
W(b)    vector_fiq

像这样我可以识别一些文件,如

entry-armv.S
timerqueue.c
thread_info.h
spinlock.c

和他们的台词。有了这个我怎么知道内核或内存出了什么问题?

2)如果恐慌发生时内核恐慌日志指向上述文件我们可以从中弄清楚什么?我们能找到内存损坏的原因以及这些信息的原因吗?

内核恐慌日志

[ 6814.239517] Unable to handle kernel NULL pointer dereference at virtual 
address 00000408
[ 6814.247646] pgd = 80004000
[ 6814.250364] [00000408] *pgd=00000000
[ 6814.253979] Internal error: Oops: 17 [#1] PREEMPT SMP ARM
[ 6814.259387] Modules linked in: mxc_v4l2_capture ipu_bg_overlay_sdc tw6869 
videobuf2_dma_contig ipu_still ipu_prp_enc videobuf2_memops ipu_csi_enc 
adv7610_video ipu_fg_overlay_sdc v4l2_int_device galcore(O)[ 6814.276790] 
DEBUG: ADV7610 HDMI cable detected TENGRI-RC
[ 6814.280811] DEBUG: ADV7610 read 1280x720p
[ 6814.282531] DEBUG: ADV7610 returned 1280x720p@61

[ 6814.291626]
[ 6814.293327] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G O 4.1.15-2.0.0- 
ga+yocto+gff4e28b #1
[ 6814.302558] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[ 6814.309096] task: ce11b200 ti: ce14a000 task.ti: ce14a000
[ 6814.314520] PC is at rb_next+0x2c/0x6c
[ 6814.318282] LR is at timerqueue_del+0x48/0x80
[ 6814.322647] pc : [<804a47e8>] lr : [<804a7128>] psr: 20030193
[ 6814.322647] sp : ce14bd80 ip : ce14bd90 fp : ce14bd8c
[ 6814.334128] r10: 00000002 r9 : 00000001 r8 : ce14be40
[ 6814.339356] r7 : d0f0f650 r6 : 00000000 r5 : d0f0f404 r4 : d0f0f650
[ 6814.345887] r3 : 00000400 r2 : d0f0f650 r1 : d0f0f650 r0 : 00000400
[ 6814.352419] Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment 
kernel
[ 6814.359819] Control: 10c5387d Table: 5e96404a DAC: 00000015
[ 6814.365569] Process swapper/1 (pid: 0, stack limit = 0xce14a210)
[ 6814.371580] Stack: (0xce14bd80 to 0xce14c000)
[ 6814.375945] bd80: ce14bdac ce14bd90 804a7128 804a47c8 d0f0f650 d0f0f3f8 
00000000 d0f0f650
[ 6814.384128] bda0: ce14bddc ce14bdb0 8018fa6c 804a70ec 00000000 d0f0f650 
00000632 d0f0f3f8
[ 6814.392311] bdc0: d0f0f3c0 ce14be40 00000001 d0f0f4d8 ce14be04 ce14bde0 
8019025c 8018fa30
[ 6814.400494] bde0: 8a653461 00000632 d0f0f3f8 00000000 d0f0f3c0 00000001 
ce14be7c ce14be08
[ 6814.408677] be00: 80190894 80190210 8a653461 00000632 80e02508 d0f0f4d0 
d0f0f498 d0f0f460
[ 6814.416860] be20: 00000003 d0f0f3f8 ffffffff 7fffffff 8a653461 00000632 
8a653461 00000632
[ 6814.425042] be40: 8a653461 00000632 ce14be6c dc8ba30f 00008648 80d98648 
80d99c34 00000001
[ 6814.433225] be60: 80e88b48 00000000 00000001 80e02508 ce14be94 ce14be80 
801a061c 80190768
[ 6814.441407] be80: 80d99c34 80d99c34 ce14bec4 ce14be98 8010ff50 801a05e0 
ce14bee8 f4a0010c
[ 6814.449590] bea0: 80e02f7c ce14bee8 f4a00100 d0f11ed0 80e08b14 80e02508 
ce14bee4 ce14bec8
[ 6814.457772] bec0: 80101594 8010fd60 806fb0b0 60030013 ffffffff ce14bf1c 
ce14bf8c ce14bee8
[ 6814.465955] bee0: 8010d240 80101538 00000000 80ed4c20 dc8ba30f dc8ba30f 
89d010b1 00000632
[ 6814.474137] bf00: 80e89160 00000004 d0f11ed0 80e08b14 80e02508 ce14bf8c 
ce14bed0 ce14bf30
[ 6814.482320] bf20: 80944994 806fb0b0 60030013 ffffffff 8a652374 00000632 
89d010b1 00000632
[ 6814.490502] bf40: 00000004 00000000 0094f441 00000001 8a652374 00000632 
00000000 dc8ba30f
[ 6814.498685] bf60: 80d97320 ce14a000 80e025e8 80e89160 d0f11ed0 80e08a74 
80a02970 ce14bfa0
[ 6814.506868] bf80: ce14bf9c ce14bf90 806fb390 806faffc ce14bfdc ce14bfa0 
80171a74 806fb378
[ 6814.515050] bfa0: 80d9b880 80d9b880 00000000 80e02e00 80e025f0 80e888ea 
00000001 80c137fc
[ 6814.523233] bfc0: 80d9aec8 80d97300 10c0387d 80e8e35c ce14bff4 ce14bfe0 
8010fae0 801717d8
[ 6814.531416] bfe0: 5e13006a 00000015 00000000 ce14bff8 1010162c 8010f994 
5b3c779b 85ac6ed7
[ 6814.539594] Backtrace:
[ 6814.542068] [<804a47bc>] (rb_next) from [<804a7128>] 
(timerqueue_del+0x48/0x80)
[ 6814.549393] [<804a70e0>] (timerqueue_del) from [<8018fa6c>] 
(__remove_hrtimer+0x48/0xdc)
[ 6814.557487] r7:d0f0f650 r6:00000000 r5:d0f0f3f8 r4:d0f0f650
[ 6814.563218] [<8018fa24>] (__remove_hrtimer) from [<8019025c>] 
(__run_hrtimer+0x58/0x284)
[ 6814.571310] r10:d0f0f4d8 r9:00000001 r8:ce14be40 r7:d0f0f3c0 r6:d0f0f3f8 
r5:00000632
[ 6814.579220] r4:d0f0f650 r3:00000000
[ 6814.582837] [<80190204>] (__run_hrtimer) from [<80190894>] 
(hrtimer_interrupt+0x138/0x344)
[ 6814.591104] r9:00000001 r8:d0f0f3c0 r7:00000000 r6:d0f0f3f8 r5:00000632 
r4:8a653461
[ 6814.598940] [<8019075c>] (hrtimer_interrupt) from [<801a061c>] 
(tick_receive_broadcast+0x48/0x60)
[ 6814.607813] r10:80e02508 r9:00000001 r8:00000000 r7:80e88b48 r6:00000001 
r5:80d99c34
[ 6814.615721] r4:80d98648
[ 6814.618283] [<801a05d4>] (tick_receive_broadcast) from [<8010ff50>] 
(handle_IPI+0x1fc/0x2f8)
[ 6814.626723] r5:80d99c34 r4:80d99c34
[ 6814.630337] [<8010fd54>] (handle_IPI) from [<80101594>] 
(gic_handle_irq+0x68/0x6c)
[ 6814.637909] r10:80e02508 r9:80e08b14 r8:d0f11ed0 r7:f4a00100 r6:ce14bee8 
r5:80e02f7c
[ 6814.645818] r4:f4a0010c r3:ce14bee8
[ 6814.649431] [<8010152c>] (gic_handle_irq) from [<8010d240>] 
(__irq_svc+0x40/0x74)
[ 6814.656917] Exception stack(0xce14bee8 to 0xce14bf30)
[ 6814.661974] bee0: 00000000 80ed4c20 dc8ba30f dc8ba30f 89d010b1 00000632
[ 6814.670157] bf00: 80e89160 00000004 d0f11ed0 80e08b14 80e02508 ce14bf8c 
ce14bed0 ce14bf30
[ 6814.678338] bf20: 80944994 806fb0b0 60030013 ffffffff
[ 6814.683392] r7:ce14bf1c r6:ffffffff r5:60030013 r4:806fb0b0
[ 6814.689129] [<806faff0>] (cpuidle_enter_state) from [<806fb390>] 
(cpuidle_enter+0x24/0x28)
[ 6814.697395] r10:ce14bfa0 r9:80a02970 r8:80e08a74 r7:d0f11ed0 r6:80e89160 
r5:80e025e8
[ 6814.705304] r4:ce14a000
[ 6814.707865] [<806fb36c>] (cpuidle_enter) from [<80171a74>] 
(cpu_startup_entry+0x2a8/0x43c)
[ 6814.716137] [<801717cc>] (cpu_startup_entry) from [<8010fae0>] 
(secondary_start_kernel+0x158/0x164)
[ 6814.725184] r7:80e8e35c
[ 6814.727743] [<8010f988>] (secondary_start_kernel) from [<1010162c>] 
(0x1010162c)
[ 6814.735142] r5:00000015 r4:5e13006a
[ 6814.738754] Code: e5923004 e3530000 0a000004 e1a00003 (e5933008)
[ 6814.744855] ---[ end trace 4a16766634a9d08f ]---
[ 6814.749478] Kernel panic - not syncing: Fatal exception in interrupt
[ 6814.755842] CPU0: stopping
[ 6814.758566] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D O 4.1.15-2.0.0- 
ga+yocto+gff4e28b #1
[ 6814.767789] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[ 6814.774320] Backtrace:
[ 6814.776799] [<8010c358>] (dump_backtrace) from [<8010c5d4>] 
(show_stack+0x20/0x24)
[ 6814.784372] r7:80e88b48 r6:80e45d94 r5:00000000 r4:80e45d94
[ 6814.790107] [<8010c5b4>] (show_stack) from [<8093ee58>] 
(dump_stack+0x7c/0xbc)
[ 6814.797338] [<8093eddc>] (dump_stack) from [<80110038>] 
(handle_IPI+0x2e4/0x2f8)
[ 6814.804737] r7:80e88b48 r6:00000005 r5:80d99c34 r4:80d99c34
[ 6814.810469] [<8010fd54>] (handle_IPI) from [<80101594>] 
(gic_handle_irq+0x68/0x6c)
[ 6814.818041] r10:80e02508 r9:80e08b14 r8:d0f04ed0 r7:f4a00100 r6:80e01e98 
r5:80e02f7c
[ 6814.825955] r4:f4a0010c r3:80e01e98
[ 6814.829571] [<8010152c>] (gic_handle_irq) from [<8010d240>] 
(__irq_svc+0x40/0x74)
[ 6814.837057] Exception stack(0x80e01e98 to 0x80e01ee0)
[ 6814.842114] 1e80: 00000000 d0f084c0
[ 6814.850299] 1ea0: dc8ba30f dc8ba30f a8cb74e4 00000632 80e89160 00000004 
d0f04ed0 80e08b14
[ 6814.858483] 1ec0: 80e02508 80e01f3c 80e01e80 80e01ee0 80944994 806fb0b0 
600f0013 ffffffff
[ 6814.866662] r7:80e01ecc r6:ffffffff r5:600f0013 r4:806fb0b0
[ 6814.872399] [<806faff0>] (cpuidle_enter_state) from [<806fb390>] 
(cpuidle_enter+0x24/0x28)
[ 6814.880665] r10:80e01f50 r9:80a02970 r8:80e08a74 r7:d0f04ed0 r6:80e89160 
r5:80e025e8
[ 6814.888581] r4:80e00000
[ 6814.891140] [<806fb36c>] (cpuidle_enter) from [<80171a74>] 
(cpu_startup_entry+0x2a8/0x43c)
[ 6814.899418] [<801717cc>] (cpu_startup_entry) from [<8093cb20>] 
(rest_init+0x98/0x9c)
[ 6814.907165] r7:80e02500
[ 6814.909731] [<8093ca88>] (rest_init) from [<80d00d9c>] 
(start_kernel+0x404/0x424)
[ 6814.917216] r5:80e8e000 r4:80e8e04c
[ 6814.920833] [<80d00998>] (start_kernel) from [<1000807c>] (0x1000807c)
[ 6814.927366] CPU3: stopping
[ 6814.930085] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D O 4.1.15-2.0.0- 
ga+yocto+gff4e28b #1
[ 6814.939309] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[ 6814.945839] Backtrace:
[ 6814.948315] [<8010c358>] (dump_backtrace) from [<8010c5d4>] 
(show_stack+0x20/0x24)
[ 6814.955888] r7:80e88b48 r6:80e45d94 r5:00000000 r4:80e45d94
[ 6814.961621] [<8010c5b4>] (show_stack) from [<8093ee58>] 
(dump_stack+0x7c/0xbc)
[ 6814.968852] [<8093eddc>] (dump_stack) from [<80110038>] 
(handle_IPI+0x2e4/0x2f8)
[ 6814.976250] r7:80e88b48 r6:00000005 r5:80d99c34 r4:80d99c34
[ 6814.981983] [<8010fd54>] (handle_IPI) from [<80101594>] 
(gic_handle_irq+0x68/0x6c)
[ 6814.989555] r10:80e02508 r9:80e08b14 r8:d0f2bed0 r7:f4a00100 r6:ce14fee8 
r5:80e02f7c
[ 6814.997471] r4:f4a0010c r3:ce14fee8
[ 6815.001087] [<8010152c>] (gic_handle_irq) from [<8010d240>] 
(__irq_svc+0x40/0x74)
[ 6815.008572] Exception stack(0xce14fee8 to 0xce14ff30)
[ 6815.013631] fee0: 00000000 d0f2f4c0 dc8ba30f dc8ba30f a8cc0d3c 00000632
[ 6815.021815] ff00: 80e89160 00000004 d0f2bed0 80e08b14 80e02508 ce14ff8c 
ce14fed0 ce14ff30
[ 6815.029996] ff20: 80944994 806fb0b0 60010013 ffffffff
[ 6815.035050] r7:ce14ff1c r6:ffffffff r5:60010013 r4:806fb0b0
[ 6815.040786] [<806faff0>] (cpuidle_enter_state) from [<806fb390>] 
(cpuidle_enter+0x24/0x28)
[ 6815.049052] r10:ce14ffa0 r9:80a02970 r8:80e08a74 r7:d0f2bed0 r6:80e89160 
r5:80e025e8
[ 6815.056967] r4:ce14e000
[ 6815.059525] [<806fb36c>] (cpuidle_enter) from [<80171a74>] 
(cpu_startup_entry+0x2a8/0x43c)
[ 6815.067798] [<801717cc>] (cpu_startup_entry) from [<8010fae0>] 
(secondary_start_kernel+0x158/0x164)
[ 6815.076846] r7:80e8e35c
[ 6815.079404] [<8010f988>] (secondary_start_kernel) from [<1010162c>] 
(0x1010162c)
[ 6815.086803] r5:00000015 r4:5e13006a
[ 6815.090414] CPU2: stopping
[ 6815.093135] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D O 4.1.15-2.0.0- 
ga+yocto+gff4e28b #1
[ 6815.102358] Hardware name: Freescale i.MX6 Quad/DualLite (Device Tree)
[ 6815.108888] Backtrace:
[ 6815.111364] [<8010c358>] (dump_backtrace) from [<8010c5d4>] 
(show_stack+0x20/0x24)
[ 6815.118937] r7:80e88b48 r6:80e45d94 r5:00000000 r4:80e45d94
[ 6815.124669] [<8010c5b4>] (show_stack) from [<8093ee58>] 
(dump_stack+0x7c/0xbc)
[ 6815.131901] [<8093eddc>] (dump_stack) from [<80110038>] 
(handle_IPI+0x2e4/0x2f8)
[ 6815.139300] r7:80e88b48 r6:00000005 r5:80d99c34 r4:80d99c34
[ 6815.145032] [<8010fd54>] (handle_IPI) from [<80101594>] 
(gic_handle_irq+0x68/0x6c)
[ 6815.152604] r10:80e02508 r9:80e08b14 r8:d0f1eed0 r7:f4a00100 r6:ce14dee8 
r5:80e02f7c
[ 6815.160519] r4:f4a0010c r3:ce14dee8
[ 6815.164134] [<8010152c>] (gic_handle_irq) from [<8010d240>] 
(__irq_svc+0x40/0x74)
[ 6815.171620] Exception stack(0xce14dee8 to 0xce14df30)
[ 6815.176679] dee0: 00000000 d0f224c0 dc8ba30f dc8ba30f a87f549c 00000632
[ 6815.184863] df00: 80e89160 00000004 d0f1eed0 80e08b14 80e02508 ce14df8c 
ce14ded0 ce14df30
[ 6815.193044] df20: 80944994 806fb0b0 60010013 ffffffff
[ 6815.198099] r7:ce14df1c r6:ffffffff r5:60010013 r4:806fb0b0
[ 6815.203834] [<806faff0>] (cpuidle_enter_state) from [<806fb390>] 
(cpuidle_enter+0x24/0x28)
[ 6815.212100] r10:ce14dfa0 r9:80a02970 r8:80e08a74 r7:d0f1eed0 r6:80e89160 
r5:80e025e8
[ 6815.220015] r4:ce14c000
[ 6815.222574] [<806fb36c>] (cpuidle_enter) from [<80171a74>] 
(cpu_startup_entry+0x2a8/0x43c)
[ 6815.230847] [<801717cc>] (cpu_startup_entry) from [<8010fae0>] 
(secondary_start_kernel+0x158/0x164)
[ 6815.239895] r7:80e8e35c
[ 6815.242453] [<8010f988>] (secondary_start_kernel) from [<1010162c>] 
(0x1010162c)
[ 6815.249852] r5:00000015 r4:5e13006a
[ 6815.253467] ---[ end Kernel panic - not syncing: Fatal exception in 
interrupt

此致 彼得。

2 个答案:

答案 0 :(得分:1)

从源代码中,rb_next传递一个指向树节点的指针。从你的恐慌日志:

  

[6814.345887] r3:00000400 r2:d0f0f650 r1:d0f0f650 r0:00000400

在ARM上,r0保存第一个参数,它看起来不像有效地址。请参阅导致OOPS的取消引用为0x00000408,因此 - 正如预期的那样 - 它在

while (node->rb_left) /* Here node == 0x00000408, which is `passed_node->rb_right` */

下一步,获取一个coredump并检查有问题的计时器(来自timerqueue_del)。

触发此操作的程序可能会告诉我们更多信息。

答案 1 :(得分:0)

我认为这不是硬件问题,因为取消引用发生在虚拟内存而非物理内容中。

您可以解码您的Ooops以获取更多信息:

17它的十进制值你得到你的10001之后需要将它转换为二进制

您可以在arch/arch/mm/fault.c

找到更详细的信息

从右到左开始解码:

bit 0 ==    0: no page found                 1: protection fault
bit 1 ==    0: read access                   1: write access
bit 2 ==    0: kernel-mode access            1: user-mode access
bit 3 ==    0: reserved bit not detect       1: use of reserved bit detected
bit 4 ==    0: No fault at instuction fetch  1: fault was an instruction fetch

在你的情况下:

Protection fault | read access | kernel | bit not set | fault on instruction

所以添加一些调试打印并检查NULL指针。