My system is a embedded linux system(running kernel version 2.6.18). A client process send data to mysql server. The data will be stored in mysql database at a RAID5 assembled by four disks. The IO pressure(wa%) is always above 20% , mysql CPU utilization is very high.
运行5或6小时后,系统会进入软锁状态。 堆栈信息是关于释放物理内存,将缓存数据写入硬盘。
在这种情况下有什么建议吗?**
BUG: soft lockup detected on CPU#0!
[<c043dc1c>] softlockup_tick+0x8f/0xb1
[<c0428cb5>] update_process_times+0x26/0x5c
[<c0411256>] smp_apic_timer_interrupt+0x5d/0x67
[<c04044e7>] apic_timer_interrupt+0x1f/0x24
[<c06fe0b9>] _spin_lock+0x5/0xf
[<c047db2a>] __mark_inode_dirty+0x50/0x176
[<c0424eef>] current_fs_time+0x4d/0x5e
[<c0475ccd>] touch_atime+0x51/0x94
[<c0440926>] do_generic_mapping_read+0x425/0x563
[<c044134b>] __generic_file_aio_read+0xf3/0x267
[<c043fcd0>] file_read_actor+0x0/0xd4
[<c04414fb>] generic_file_aio_read+0x3c/0x4d
[<c045d72d>] do_sync_read+0xc1/0xfd
[<c0431656>] autoremove_wake_function+0x0/0x37
[<c045e0e8>] vfs_read+0xa4/0x167
[<c045d66c>] do_sync_read+0x0/0xfd
[<c045e688>] sys_pread64+0x5e/0x62
[<c0403a27>] syscall_call+0x7/0xb
=======================
BUG: soft lockup detected on CPU#2!
[<c043dc1c>] softlockup_tick+0x8f/0xb1
[<c0428cb5>] update_process_times+0x26/0x5c
[<c0411256>] smp_apic_timer_interrupt+0x5d/0x67
[<c04044e7>] apic_timer_interrupt+0x1f/0x24
[<c06fe0bb>] _spin_lock+0x7/0xf
[<c04aaf17>] journal_try_to_free_buffers+0xf4/0x17b
[<c0442c52>] find_get_pages+0x28/0x5d
[<c049c4b1>] ext3_releasepage+0x0/0x7d
[<c045f0bf>] try_to_release_page+0x2c/0x46
[<c0447894>] invalidate_mapping_pages+0xc9/0x167
[<c04813b0>] drop_pagecache+0x86/0xd2
[<c048144e>] drop_caches_sysctl_handler+0x52/0x64
[<c04813fc>] drop_caches_sysctl_handler+0x0/0x64
[<c042623d>] do_rw_proc+0xe8/0xf4
[<c0426268>] proc_writesys+0x1f/0x24
[<c045df81>] vfs_write+0xa6/0x169
[<c0426249>] proc_writesys+0x0/0x24
[<c045e601>] sys_write+0x41/0x6a
[<c0403a27>] syscall_call+0x7/0xb
=======================
BUG: soft lockup detected on CPU#1!
[<c043dc1c>] softlockup_tick+0x8f/0xb1
[<c0428cb5>] update_process_times+0x26/0x5c
[<c0411256>] smp_apic_timer_interrupt+0x5d/0x67
[<c04044e7>] apic_timer_interrupt+0x1f/0x24
[<c06f007b>] inet_diag_dump+0x804/0x821
[<c06fe0bb>] _spin_lock+0x7/0xf
[<c047db2a>] __mark_inode_dirty+0x50/0x176
[<c043168d>] wake_bit_function+0x0/0x3c
[<c04ae0f6>] __journal_remove_journal_head+0xee/0x1a5
[<c0445ae8>] __set_page_dirty_nobuffers+0x87/0xc6
[<c04a908e>] __journal_unfile_buffer+0x8/0x11
[<c04ab94d>] journal_commit_transaction+0x8e0/0x1103
[<c0431656>] autoremove_wake_function+0x0/0x37
[<c04af690>] kjournald+0xa9/0x1e5
[<c0431656>] autoremove_wake_function+0x0/0x37
[<c04af5e7>] kjournald+0x0/0x1e5
[<c04314da>] kthread+0xde/0xe2
[<c04313fc>] kthread+0x0/0xe2
[<c0404763>] kernel_thread_helper+0x7/0x14
=======================
BUG: soft lockup detected on CPU#3!
[<c043dc1c>] softlockup_tick+0x8f/0xb1
[<c0428cb5>] update_process_times+0x26/0x5c
[<c0411256>] smp_apic_timer_interrupt+0x5d/0x67
[<c04044e7>] apic_timer_interrupt+0x1f/0x24
[<c06fe0bb>] _spin_lock+0x7/0xf
[<c047db2a>] __mark_inode_dirty+0x50/0x176
[<c0424eef>] current_fs_time+0x4d/0x5e
[<c0475ccd>] touch_atime+0x51/0x94
[<c0440926>] do_generic_mapping_read+0x425/0x563
[<c044134b>] __generic_file_aio_read+0xf3/0x267
[<c043fcd0>] file_read_actor+0x0/0xd4
[<c04414fb>] generic_file_aio_read+0x3c/0x4d
[<c045d72d>] do_sync_read+0xc1/0xfd
[<c0431656>] autoremove_wake_function+0x0/0x37
[<c045e0e8>] vfs_read+0xa4/0x167
[<c045d66c>] do_sync_read+0x0/0xfd
[<c045e688>] sys_pread64+0x5e/0x62
[<c0403a27>] syscall_call+0x7/0xb
=======================
答案 0 :(得分:0)
说真的,尝试更新的东西。 2.6.18>> 7岁。
答案 1 :(得分:0)
看起来CPU#1和CPU#3正在inode结构中的自旋锁中旋转。