令我困惑的是,有大型非活动文件页面缓存734812kB,脏缓存800088kB似乎可以回收,为什么oom-killer会发生?
vm.swappiness设置为0,如linux文档中所述,0并不意味着完全避免交换,并且我发现使用的交换空间达到300MB,swappiness = 0,其他服务器上的设置和操作系统相同。
操作系统信息:
CentOS 6.4 kernel: 2.6.32-358.23.2.el6.x86_64
oom-killer日志如下:
Aug 26 14:34:48 withivan.me kernel: java invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
Aug 26 14:34:48 withivan.me kernel: java cpuset=/ mems_allowed=0
Aug 26 14:34:48 withivan.me kernel: Pid: 28505, comm: java Not tainted 2.6.32-358.23.2.el6.x86_64 #1
Aug 26 14:34:48 withivan.me kernel: Call Trace:
Aug 26 14:34:48 withivan.me kernel: [<ffffffff810cb641>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8111ce40>] ? dump_header+0x90/0x1b0
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8121d4ec>] ? security_real_capable_noaudit+0x3c/0x70
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8111d2c2>] ? oom_kill_process+0x82/0x2a0
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8111d201>] ? select_bad_process+0xe1/0x120
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8111d700>] ? out_of_memory+0x220/0x3c0
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8112c3dc>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Aug 26 14:34:48 withivan.me kernel: [<ffffffff81160c6a>] ? alloc_pages_current+0xaa/0x110
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8148d667>] ? tcp_sendmsg+0x677/0xa20
Aug 26 14:34:48 withivan.me kernel: [<ffffffff81435f33>] ? sock_sendmsg+0x123/0x150
Aug 26 14:34:48 withivan.me kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Aug 26 14:34:48 withivan.me kernel: [<ffffffff810aa43e>] ? futex_wake+0x10e/0x120
Aug 26 14:34:48 withivan.me kernel: [<ffffffff810ac3a0>] ? do_futex+0x100/0xb60
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8119cfdf>] ? destroy_inode+0x2f/0x60
Aug 26 14:34:48 withivan.me kernel: [<ffffffff81436249>] ? sys_sendto+0x139/0x190
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8103b8cc>] ? kvm_clock_read+0x1c/0x20
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8103b8d9>] ? kvm_clock_get_cycles+0x9/0x10
Aug 26 14:34:48 withivan.me kernel: [<ffffffff810a1507>] ? getnstimeofday+0x57/0xe0
Aug 26 14:34:48 withivan.me kernel: [<ffffffff810a15fa>] ? do_gettimeofday+0x1a/0x50
Aug 26 14:34:48 withivan.me kernel: [<ffffffff8100b072>] ? system_call_fastpath+0x16/0x1b
Aug 26 14:34:48 withivan.me kernel: Mem-Info:
Aug 26 14:34:48 withivan.me kernel: Node 0 DMA per-cpu:
Aug 26 14:34:48 withivan.me kernel: CPU 0: hi: 0, btch: 1 usd: 0
Aug 26 14:34:48 withivan.me kernel: CPU 1: hi: 0, btch: 1 usd: 0
Aug 26 14:34:48 withivan.me kernel: CPU 2: hi: 0, btch: 1 usd: 0
Aug 26 14:34:48 withivan.me kernel: CPU 3: hi: 0, btch: 1 usd: 0
Aug 26 14:34:48 withivan.me kernel: Node 0 DMA32 per-cpu:
Aug 26 14:34:48 withivan.me kernel: CPU 0: hi: 186, btch: 31 usd: 32
Aug 26 14:34:48 withivan.me kernel: CPU 1: hi: 186, btch: 31 usd: 0
Aug 26 14:34:48 withivan.me kernel: CPU 2: hi: 186, btch: 31 usd: 0
Aug 26 14:34:48 withivan.me kernel: CPU 3: hi: 186, btch: 31 usd: 1
Aug 26 14:34:48 withivan.me kernel: Node 0 Normal per-cpu:
Aug 26 14:34:48 withivan.me kernel: CPU 0: hi: 186, btch: 31 usd: 4
Aug 26 14:34:48 withivan.me kernel: CPU 1: hi: 186, btch: 31 usd: 38
Aug 26 14:34:48 withivan.me kernel: CPU 2: hi: 186, btch: 31 usd: 0
Aug 26 14:34:48 withivan.me kernel: CPU 3: hi: 186, btch: 31 usd: 57
Aug 26 14:34:48 withivan.me kernel: active_anon:1697553 inactive_anon:373583 isolated_anon:0
Aug 26 14:34:48 withivan.me kernel: active_file:174263 inactive_file:199171 isolated_file:0
Aug 26 14:34:48 withivan.me kernel: unevictable:0 dirty:216860 writeback:1 unstable:0
Aug 26 14:34:48 withivan.me kernel: free:35470 slab_reclaimable:14993 slab_unreclaimable:6945
Aug 26 14:34:48 withivan.me kernel: mapped:3423 shmem:45 pagetables:5263 bounce:0
Aug 26 14:34:48 withivan.me kernel: Node 0 DMA free:15740kB min:148kB low:184kB high:220kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15344kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Aug 26 14:34:48 withivan.me kernel: lowmem_reserve[]: 0 3512 10077 10077
Aug 26 14:34:48 withivan.me kernel: Node 0 DMA32 free:61200kB min:34800kB low:43500kB high:52200kB active_anon:2479864kB inactive_anon:621800kB active_file:39752kB inactive_file:61872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3596500kB mlocked:0kB dirty:67352kB writeback:0kB mapped:20kB shmem:0kB slab_reclaimable:21672kB slab_unreclaimable:1952kB kernel_stack:3792kB pagetables:4344kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:94464 all_unreclaimable? no
Aug 26 14:34:48 withivan.me kernel: lowmem_reserve[]: 0 0 6565 6565
Aug 26 14:34:48 withivan.me kernel: Node 0 Normal free:64940kB min:65048kB low:81308kB high:97572kB active_anon:4310348kB inactive_anon:872532kB active_file:657300kB inactive_file:734812kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:6722560kB mlocked:0kB dirty:800088kB writeback:4kB mapped:13672kB shmem:180kB slab_reclaimable:38300kB slab_unreclaimable:25828kB kernel_stack:4568kB pagetables:16708kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:605088 all_unreclaimable? no
Aug 26 14:34:48 withivan.me kernel: lowmem_reserve[]: 0 0 0 0
Aug 26 14:34:48 withivan.me kernel: Node 0 DMA: 3*4kB 0*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB
Aug 26 14:34:48 withivan.me kernel: Node 0 DMA32: 1247*4kB 1189*8kB 969*16kB 379*32kB 92*64kB 12*128kB 1*256kB 8*512kB 5*1024kB 1*2048kB 0*4096kB = 61076kB
Aug 26 14:34:48 withivan.me kernel: Node 0 Normal: 2047*4kB 1672*8kB 781*16kB 309*32kB 46*64kB 3*128kB 1*256kB 20*512kB 7*1024kB 0*2048kB 0*4096kB = 64940kB
Aug 26 14:34:48 withivan.me kernel: 379124 total pagecache pages
Aug 26 14:34:48 withivan.me kernel: 4685 pages in swap cache
Aug 26 14:34:48 withivan.me kernel: Swap cache stats: add 167082, delete 162397, find 114795/130707
Aug 26 14:34:48 withivan.me kernel: Free swap = 4166416kB
Aug 26 14:34:48 withivan.me kernel: Total swap = 4194296kB
Aug 26 14:34:48 withivan.me kernel: 2621439 pages RAM
Aug 26 14:34:48 withivan.me kernel: 89408 pages reserved
Aug 26 14:34:48 withivan.me kernel: 384993 pages shared
Aug 26 14:34:48 withivan.me kernel: 2116876 pages non-shared