Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1183436 > unrolled thread

Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory

Started byMichal Hocko <mhocko@kernel.org>
First post2015-07-14 10:20 +0200
Last post2015-07-14 12:00 +0200
Articles 6 — 2 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [BUG REPORT] OOM Killer is invoked while the system still has  much memory Michal Hocko <mhocko@kernel.org> - 2015-07-14 10:20 +0200
    Re: [BUG REPORT] OOM Killer is invoked while the system still has  much memory Xishi Qiu <qiuxishi@huawei.com> - 2015-07-14 10:50 +0200
      Re: [BUG REPORT] OOM Killer is invoked while the system still has  much memory Michal Hocko <mhocko@kernel.org> - 2015-07-14 11:10 +0200
        Re: [BUG REPORT] OOM Killer is invoked while the system still has  much memory Xishi Qiu <qiuxishi@huawei.com> - 2015-07-14 11:20 +0200
          Re: [BUG REPORT] OOM Killer is invoked while the system still has  much memory Michal Hocko <mhocko@kernel.org> - 2015-07-14 11:30 +0200
            Re: [BUG REPORT] OOM Killer is invoked while the system still has  much memory Xishi Qiu <qiuxishi@huawei.com> - 2015-07-14 12:00 +0200

#1183436 — Re: [BUG REPORT] OOM Killer is invoked while the system still has much memory

FromMichal Hocko <mhocko@kernel.org>
Date2015-07-14 10:20 +0200
SubjectRe: [BUG REPORT] OOM Killer is invoked while the system still has much memory
Message-ID<pM3Bw-8dT-7@gated-at.bofh.it>
On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
> Hi, all
> 
> Description of problem:
> 
> Recently, one of my Linux system invoked oom-killer, but the system
> still has much memory, I don't know why the system still invoked
> oom-killer, anybody can help me to see it, thanks.
> 
> Linux kernel version: 3.0.58
> 
> Following is the message:
> 
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
[...]
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014 3014 3014
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32 free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB inactive_anon:41528kB active_file:812kB inactive_file:756kB unevictable:381580kB isolated(anon):0kB isolated(file):188kB present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB mapped:45940kB shmem:44668kB slab_reclaimable:72748kB slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192 all_unreclaimable? no

You are well above watermarks but note that you have basically no
pages on the file LRU and you have _no swap_ so the anon memory is
unreclaimable. There is still around 72M of reclaimable slab but that
could be hard to reclaim due to internal fragmentation. The allocation
request is GFP_KERNEL so the slab shrinkers shouldn't back off due to
__GFP_FS restrictions.

> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138960] lowmem_reserve[]: 0 0 0 0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB

Moreover your allocation request was oreder 2 and you do not have much
memory there because most of the free memory is in order-0-2.

> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138974] 12622 total pagecache pages
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138976] 0 pages in swap cache
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138977] Swap cache stats: add 0, delete 0, find 0/0
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138978] Free swap  = 0kB
> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138979] Total swap = 0kB
						      ^^^^^^^^^^^^^^^^
So I am not surprised about the oom killer much.

-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1183467

FromXishi Qiu <qiuxishi@huawei.com>
Date2015-07-14 10:50 +0200
Message-ID<pM44y-8nD-11@gated-at.bofh.it>
In reply to#1183436
On 2015/7/14 16:15, Michal Hocko wrote:

> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
>> Hi, all
>>
>> Description of problem:
>>
>> Recently, one of my Linux system invoked oom-killer, but the system
>> still has much memory, I don't know why the system still invoked
>> oom-killer, anybody can help me to see it, thanks.
>>
>> Linux kernel version: 3.0.58
>>
>> Following is the message:
>>
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138764] iostat invoked oom-killer: gfp_mask=0xd0, order=2, oom_adj=0, oom_score_adj=0
> [...]
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138945] DMA free:984kB min:36kB low:44kB high:52kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:16160kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138949] lowmem_reserve[]: 0 3014 3014 3014
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138955] DMA32 free:990352kB min:7004kB low:8752kB high:10504kB active_anon:908444kB inactive_anon:41528kB active_file:812kB inactive_file:756kB unevictable:381580kB isolated(anon):0kB isolated(file):188kB present:3025264kB mlocked:381580kB dirty:0kB writeback:0kB mapped:45940kB shmem:44668kB slab_reclaimable:72748kB slab_unreclaimable:215412kB kernel_stack:12456kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:192 all_unreclaimable? no
> 
> You are well above watermarks but note that you have basically no
> pages on the file LRU and you have _no swap_ so the anon memory is
> unreclaimable. There is still around 72M of reclaimable slab but that
> could be hard to reclaim due to internal fragmentation. The allocation
> request is GFP_KERNEL so the slab shrinkers shouldn't back off due to
> __GFP_FS restrictions.
> 
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138960] lowmem_reserve[]: 0 0 0 0
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138962] DMA: 2*4kB 4*8kB 3*16kB 4*32kB 2*64kB 1*128kB 2*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 984kB
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
> 
> Moreover your allocation request was oreder 2 and you do not have much
> memory there because most of the free memory is in order-0-2.
> 

Hi Michal,

order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB, 
so you mean this large buddy block was reclaimed during the moment of oom and 
print, right?

Thanks,
Xishi Qiu

>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138974] 12622 total pagecache pages
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138976] 0 pages in swap cache
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138977] Swap cache stats: add 0, delete 0, find 0/0
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138978] Free swap  = 0kB
>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138979] Total swap = 0kB
> 						      ^^^^^^^^^^^^^^^^
> So I am not surprised about the oom killer much.
> 



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1183475

FromMichal Hocko <mhocko@kernel.org>
Date2015-07-14 11:10 +0200
Message-ID<pM4nU-hK-3@gated-at.bofh.it>
In reply to#1183467
On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
> On 2015/7/14 16:15, Michal Hocko wrote:
> 
> > On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
[...]
> >> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
> > 
> > Moreover your allocation request was oreder 2 and you do not have much
> > memory there because most of the free memory is in order-0-2.
> > 
> 
> Hi Michal,
> 
> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB, 
> so you mean this large buddy block was reclaimed during the moment of oom and 
> print, right?

Not really. Those high order blocks are inaccessible for your GFP_KERNEL
allocation. See __zone_watermark_ok.

-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1183487

FromXishi Qiu <qiuxishi@huawei.com>
Date2015-07-14 11:20 +0200
Message-ID<pM4xA-lo-19@gated-at.bofh.it>
In reply to#1183475
On 2015/7/14 17:00, Michal Hocko wrote:

> On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
>> On 2015/7/14 16:15, Michal Hocko wrote:
>>
>>> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
> [...]
>>>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
>>>
>>> Moreover your allocation request was oreder 2 and you do not have much
>>> memory there because most of the free memory is in order-0-2.
>>>
>>
>> Hi Michal,
>>
>> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB, 
>> so you mean this large buddy block was reclaimed during the moment of oom and 
>> print, right?
> 
> Not really. Those high order blocks are inaccessible for your GFP_KERNEL
> allocation. See __zone_watermark_ok.
> 

I know, some of them are from reserved memory(MIGRATE_RESERVE), right?

Thanks,
Xishi Qiu

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1183494

FromMichal Hocko <mhocko@kernel.org>
Date2015-07-14 11:30 +0200
Message-ID<pM4Hh-oF-39@gated-at.bofh.it>
In reply to#1183487
On Tue 14-07-15 17:14:40, Xishi Qiu wrote:
> On 2015/7/14 17:00, Michal Hocko wrote:
> 
> > On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
> >> On 2015/7/14 16:15, Michal Hocko wrote:
> >>
> >>> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
> > [...]
> >>>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
> >>>
> >>> Moreover your allocation request was oreder 2 and you do not have much
> >>> memory there because most of the free memory is in order-0-2.
> >>>
> >>
> >> Hi Michal,
> >>
> >> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB, 
> >> so you mean this large buddy block was reclaimed during the moment of oom and 
> >> print, right?
> > 
> > Not really. Those high order blocks are inaccessible for your GFP_KERNEL
> > allocation. See __zone_watermark_ok.
> > 
> 
> I know, some of them are from reserved memory(MIGRATE_RESERVE), right?

No. The watermark is calculated per order. And you have almost all the
free memory in the lower orders. From a quick glance it seems that even
order-1 allocations wouldn't fit into min watermark.
-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1183551

FromXishi Qiu <qiuxishi@huawei.com>
Date2015-07-14 12:00 +0200
Message-ID<pM5ai-zl-17@gated-at.bofh.it>
In reply to#1183494
On 2015/7/14 17:28, Michal Hocko wrote:

> On Tue 14-07-15 17:14:40, Xishi Qiu wrote:
>> On 2015/7/14 17:00, Michal Hocko wrote:
>>
>>> On Tue 14-07-15 16:42:16, Xishi Qiu wrote:
>>>> On 2015/7/14 16:15, Michal Hocko wrote:
>>>>
>>>>> On Tue 14-07-15 07:11:34, Xuzhichuang wrote:
>>> [...]
>>>>>> Jul 10 12:33:03 BMS_CNA04 kernel: [18136514.138968] DMA32: 188513*4kB 29459*8kB 2*16kB 2*32kB 1*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 990396kB
>>>>>
>>>>> Moreover your allocation request was oreder 2 and you do not have much
>>>>> memory there because most of the free memory is in order-0-2.
>>>>>
>>>>
>>>> Hi Michal,
>>>>
>>>> order=2 -> alloc 16kb memory, and DMA32 still has 2*16kB 2*32kB 1*64kB 1*512kB, 
>>>> so you mean this large buddy block was reclaimed during the moment of oom and 
>>>> print, right?
>>>
>>> Not really. Those high order blocks are inaccessible for your GFP_KERNEL
>>> allocation. See __zone_watermark_ok.
>>>
>>
>> I know, some of them are from reserved memory(MIGRATE_RESERVE), right?
> 
> No. The watermark is calculated per order. And you have almost all the
> free memory in the lower orders. From a quick glance it seems that even
> order-1 allocations wouldn't fit into min watermark.

So we can't alloc memory because it doesn't fit the watermark, and
the large buddy blocks maybe from reserved, or maybe not, right?

Thanks,
Xishi Qiu

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web