Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1588302 > unrolled thread

Re: Still OOM problems with 4.9er/4.10er kernels

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Still OOM problems with 4.9er/4.10er kernels Gerhard Wiesinger <lists@wiesinger.com> - 2017-02-26 09:50 +0100
    Re: Still OOM problems with 4.9er/4.10er kernels Michal Hocko <mhocko@kernel.org> - 2017-02-27 09:30 +0100
      Re: Still OOM problems with 4.9er/4.10er kernels Gerhard Wiesinger <lists@wiesinger.com> - 2017-02-28 07:20 +0100
        Re: Still OOM problems with 4.9er/4.10er kernels Michal Hocko <mhocko@kernel.org> - 2017-02-28 09:20 +0100
    Re: Still OOM problems with 4.9er/4.10er kernels Minchan Kim <minchan@kernel.org> - 2017-02-27 10:20 +0100
      Re: Still OOM problems with 4.9er/4.10er kernels Michal Hocko <mhocko@kernel.org> - 2017-02-27 11:10 +0100
        Re: Still OOM problems with 4.9er/4.10er kernels Minchan Kim <minchan@kernel.org> - 2017-02-28 06:50 +0100
          Re: Still OOM problems with 4.9er/4.10er kernels Michal Hocko <mhocko@kernel.org> - 2017-02-28 09:30 +0100

#1588302 — Re: Still OOM problems with 4.9er/4.10er kernels

On 04.01.2017 10:11, Michal Hocko wrote:
>> The VM stops working (e.g. not pingable) after around 8h (will be restarted
>> automatically), happened serveral times.
>>
>> Had also further OOMs which I sent to Mincham.
> Could you post them to the mailing list as well, please?

Still OOMs on dnf update procedure with kernel 4.10: 
4.10.0-1.fc26.x86_64 as well on 4.9.9-200.fc25.x86_64

On 4.10er kernels:

Free swap  = 1137532kB

cat /etc/sysctl.d/* | grep ^vm
vm.dirty_background_ratio = 3
vm.dirty_ratio = 15
vm.overcommit_memory = 2
vm.overcommit_ratio = 80
vm.swappiness=10

kernel: python invoked oom-killer: 
gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, 
order=0, oom_score_adj=0
kernel: python cpuset=/ mems_allowed=0
kernel: CPU: 1 PID: 813 Comm: python Not tainted 4.10.0-1.fc26.x86_64 #1
kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
1.9.3 04/01/2014
kernel: Call Trace:
kernel:  dump_stack+0x63/0x84
kernel:  dump_header+0x7b/0x1f6
kernel:  ? do_try_to_free_pages+0x2c5/0x340
kernel:  oom_kill_process+0x202/0x3d0
kernel:  out_of_memory+0x2b7/0x4e0
kernel:  __alloc_pages_slowpath+0x915/0xb80
kernel:  __alloc_pages_nodemask+0x218/0x2d0
kernel:  alloc_pages_current+0x93/0x150
kernel:  __page_cache_alloc+0xcf/0x100
kernel:  filemap_fault+0x39d/0x800
kernel:  ? page_add_file_rmap+0xe5/0x200
kernel:  ? filemap_map_pages+0x2e1/0x4e0
kernel:  ext4_filemap_fault+0x36/0x50
kernel:  __do_fault+0x21/0x110
kernel:  handle_mm_fault+0xdd1/0x1410
kernel:  ? swake_up+0x42/0x50
kernel:  __do_page_fault+0x23f/0x4c0
kernel:  trace_do_page_fault+0x41/0x120
kernel:  do_async_page_fault+0x51/0xa0
kernel:  async_page_fault+0x28/0x30
kernel: RIP: 0033:0x7f0681ad6350
kernel: RSP: 002b:00007ffcbdd238d8 EFLAGS: 00010246
kernel: RAX: 00007f0681b0f960 RBX: 0000000000000000 RCX: 7fffffffffffffff
kernel: RDX: 0000000000000000 RSI: 3ff0000000000000 RDI: 3ff0000000000000
kernel: RBP: 00007f067461ab40 R08: 0000000000000000 R09: 3ff0000000000000
kernel: R10: 0000556f1c6d8a80 R11: 0000000000000001 R12: 00007f0676d1a8d0
kernel: R13: 0000000000000000 R14: 00007f06746168bc R15: 00007f0674385910
kernel: Mem-Info:
kernel: active_anon:37423 inactive_anon:37512 isolated_anon:0
          active_file:462 inactive_file:603 isolated_file:0
          unevictable:0 dirty:0 writeback:0 unstable:0
          slab_reclaimable:3538 slab_unreclaimable:4818
          mapped:859 shmem:9 pagetables:3370 bounce:0
          free:1650 free_pcp:103 free_cma:0
kernel: Node 0 active_anon:149380kB inactive_anon:149704kB 
active_file:1848kB inactive_file:3660kB unevictable:0kB 
isolated(anon):128kB isolated(file):0kB mapped:4580kB dirty:0kB 
writeback:380kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 
36kB writeback_tmp:0kB unstable:0kB pages_scanned:352 all_unreclaimable? no
kernel: Node 0 DMA free:1484kB min:104kB low:128kB high:152kB 
active_anon:5660kB inactive_anon:6156kB active_file:56kB 
inactive_file:64kB unevictable:0kB writepending:0kB present:15992kB 
managed:15908kB mlocked:0kB slab_reclaimable:444kB 
slab_unreclaimable:1208kB kernel_stack:32kB pagetables:592kB bounce:0kB 
free_pcp:0kB local_pcp:0kB free_cma:0kB
kernel: lowmem_reserve[]: 0 327 327 327 327
kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB 
active_anon:143580kB inactive_anon:143300kB active_file:2576kB 
inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB 
managed:353968kB mlocked:0kB slab_reclaimable:13708kB 
slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB 
bounce:0kB free_pcp:412kB local_pcp:88kB free_cma:0kB
kernel: lowmem_reserve[]: 0 0 0 0 0
kernel: Node 0 DMA: 70*4kB (UMEH) 20*8kB (UMEH) 13*16kB (MH) 5*32kB (H) 
4*64kB (H) 2*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 
1576kB
kernel: Node 0 DMA32: 1134*4kB (UMEH) 25*8kB (UMEH) 13*16kB (MH) 7*32kB 
(H) 3*64kB (H) 0*128kB 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 
5616kB
kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 
hugepages_size=2048kB
kernel: 6561 total pagecache pages
kernel: 5240 pages in swap cache
kernel: Swap cache stats: add 100078658, delete 100073419, find 
199458343/238460223
kernel: Free swap  = 1137532kB
kernel: Total swap = 2064380kB
kernel: 98170 pages RAM
kernel: 0 pages HighMem/MovableOnly
kernel: 5701 pages reserved
kernel: 0 pages cma reserved
kernel: 0 pages hwpoisoned
kernel: Out of memory: Kill process 11968 (clamscan) score 170 or 
sacrifice child
kernel: Killed process 11968 (clamscan) total-vm:538120kB, 
anon-rss:182220kB, file-rss:464kB, shmem-rss:0kB

On 4.9er kernels:

Free swap  = 1826688kB

cat /etc/sysctl.d/* | grep ^vm
vm.dirty_background_ratio=3
vm.dirty_ratio=15
vm.overcommit_memory=2
vm.overcommit_ratio=80
vm.swappiness=10

kernel: dnf invoked oom-killer: 
gfp_mask=0x24280ca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), nodemask=0, 
order=0, oom_score_adj=0
kernel: dnf cpuset=/ mems_allowed=0
kernel: CPU: 0 PID: 20049 Comm: dnf Not tainted 4.9.9-200.fc25.x86_64 #1
kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 
1.9.3 04/01/2014
kernel:  ffffa764434d7ac0 ffffffff933f467d ffffa764434d7c90 ffff9569999057c0
kernel:  ffffa764434d7b38 ffffffff932555c1 0000000000000000 0000000000000000
kernel:  ffffffffc02b2afa 00000000ffffffff 0000000000000000 ffffa764434d7b28
kernel: Call Trace:
kernel:  [<ffffffff933f467d>] dump_stack+0x63/0x86
kernel:  [<ffffffff932555c1>] dump_header+0x7b/0x1f6
kernel:  [<ffffffffc02b2afa>] ? virtballoon_oom_notify+0x2a/0x80 
[virtio_balloon]
kernel:  [<ffffffff931c711f>] oom_kill_process+0x1ff/0x3d0
kernel:  [<ffffffff931c7623>] out_of_memory+0x143/0x4e0
kernel:  [<ffffffff931cd3cd>] __alloc_pages_slowpath+0xb2d/0xc40
kernel:  [<ffffffff931cd736>] __alloc_pages_nodemask+0x256/0x2c0
kernel:  [<ffffffff9322567b>] alloc_pages_vma+0xab/0x290
kernel:  [<ffffffff931fe2ed>] handle_mm_fault+0x13bd/0x1610
kernel:  [<ffffffff9306283e>] __do_page_fault+0x23e/0x4e0
kernel:  [<ffffffff93062ba1>] trace_do_page_fault+0x41/0x120
kernel:  [<ffffffff9305caba>] do_async_page_fault+0x1a/0xa0
kernel:  [<ffffffff9381f3b8>] async_page_fault+0x28/0x30
kernel: Mem-Info:
kernel: active_anon:31057 inactive_anon:30347 isolated_anon:0
          active_file:20333 inactive_file:25749 isolated_file:32
          unevictable:0 dirty:1444 writeback:0 unstable:0
          slab_reclaimable:4537 slab_unreclaimable:5448
          mapped:15224 shmem:120 pagetables:2537 bounce:0
          free:1133 free_pcp:196 free_cma:0
kernel: Node 0 active_anon:122692kB inactive_anon:122948kB 
active_file:81332kB inactive_file:102996kB unevictable:0kB 
isolated(anon):0kB isolated(file):128kB mapped:60896kB dirty:5776kB 
writeback:0kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 
480kB writeback_tmp:0kB unstable:0kB pages_scanned:962974 
all_unreclaimable? yes
kernel: Node 0 DMA free:1892kB min:92kB low:112kB high:132kB 
active_anon:544kB inactive_anon:10872kB active_file:16kB 
inactive_file:996kB unevictable:0kB writepending:1128kB present:15992kB 
managed:15908kB mlocked:0kB slab_reclaimable:488kB 
slab_unreclaimable:388kB kernel_stack:0kB pagetables:24kB bounce:0kB 
free_pcp:0kB local_pcp:0kB free_cma:0kB
kernel: lowmem_reserve[]: 0 451 451 451 451
kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB 
active_anon:122148kB inactive_anon:112068kB active_file:81324kB 
inactive_file:101972kB unevictable:0kB writepending:4648kB 
present:507760kB managed:484384kB mlocked:0kB slab_reclaimable:17660kB 
slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB 
bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB
kernel: lowmem_reserve[]: 0 0 0 0 0
kernel: Node 0 DMA: 81*4kB (UEH) 26*8kB (UMEH) 15*16kB (UH) 7*32kB (H) 
8*64kB (H) 3*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1892kB
kernel: Node 0 DMA32: 375*4kB (UMH) 18*8kB (MH) 7*16kB (MH) 6*32kB (MH) 
6*64kB (H) 4*128kB (H) 2*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 
3356kB
kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 
hugepages_size=2048kB
kernel: 49304 total pagecache pages
kernel: 3085 pages in swap cache
kernel: Swap cache stats: add 129095, delete 126010, find 24611247/24627536
kernel: Free swap  = 1826688kB
kernel: Total swap = 2064380kB
kernel: 130938 pages RAM
kernel: 0 pages HighMem/MovableOnly
kernel: 5865 pages reserved
kernel: 0 pages cma reserved
kernel: 0 pages hwpoisoned
kernel: Out of memory: Kill process 995 (named) score 87 or sacrifice child
kernel: Killed process 995 (named) total-vm:399260kB, anon-rss:86516kB, 
file-rss:1288kB, shmem-rss:0kB
kernel: oom_reaper: reaped process 995 (named), now anon-rss:0kB, 
file-rss:0kB, shmem-rss:0kB

Should be very easy to reproduce with a low mem VM (e.g. 192MB) under 
KVM with ext4 and Fedora 25 and some memory load and updating the VM.

Any further progress?

Thnx.

Ciao,

Gerhard

[toc] | [next] | [standalone]

#1588542

On Sun 26-02-17 09:40:42, Gerhard Wiesinger wrote:
> On 04.01.2017 10:11, Michal Hocko wrote:
> >>The VM stops working (e.g. not pingable) after around 8h (will be restarted
> >>automatically), happened serveral times.
> >>
> >>Had also further OOMs which I sent to Mincham.
> >Could you post them to the mailing list as well, please?
> 
> Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as
> well on 4.9.9-200.fc25.x86_64
> 
> On 4.10er kernels:
[...]
> kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB
> active_anon:143580kB inactive_anon:143300kB active_file:2576kB
> inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB
> managed:353968kB mlocked:0kB slab_reclaimable:13708kB
> slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB
> free_pcp:412kB local_pcp:88kB free_cma:0kB
[...]

> On 4.9er kernels:
[...]
> kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB
> active_anon:122148kB inactive_anon:112068kB active_file:81324kB
> inactive_file:101972kB unevictable:0kB writepending:4648kB present:507760kB
> managed:484384kB mlocked:0kB slab_reclaimable:17660kB
> slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB bounce:0kB
> free_pcp:120kB local_pcp:0kB free_cma:0kB

In both cases the amount if free memory is above the min watermark, so
we shouldn't be hitting the oom. We might have somebody freeing memory
after the last attempt, though...

[...]
> Should be very easy to reproduce with a low mem VM (e.g. 192MB) under KVM
> with ext4 and Fedora 25 and some memory load and updating the VM.
> 
> Any further progress?

The linux-next (resp. mmotm tree) has new tracepoints which should help
to tell us more about what is going on here. Could you try to enable
oom/reclaim_retry_zone and vmscan/mm_vmscan_direct_reclaim_{begin,end}
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1589200

On 27.02.2017 09:27, Michal Hocko wrote:
> On Sun 26-02-17 09:40:42, Gerhard Wiesinger wrote:
>> On 04.01.2017 10:11, Michal Hocko wrote:
>>>> The VM stops working (e.g. not pingable) after around 8h (will be restarted
>>>> automatically), happened serveral times.
>>>>
>>>> Had also further OOMs which I sent to Mincham.
>>> Could you post them to the mailing list as well, please?
>> Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as
>> well on 4.9.9-200.fc25.x86_64
>>
>> On 4.10er kernels:
> [...]
>> kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB
>> active_anon:143580kB inactive_anon:143300kB active_file:2576kB
>> inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB
>> managed:353968kB mlocked:0kB slab_reclaimable:13708kB
>> slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB
>> free_pcp:412kB local_pcp:88kB free_cma:0kB
> [...]
>
>> On 4.9er kernels:
> [...]
>> kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB
>> active_anon:122148kB inactive_anon:112068kB active_file:81324kB
>> inactive_file:101972kB unevictable:0kB writepending:4648kB present:507760kB
>> managed:484384kB mlocked:0kB slab_reclaimable:17660kB
>> slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB bounce:0kB
>> free_pcp:120kB local_pcp:0kB free_cma:0kB
> In both cases the amount if free memory is above the min watermark, so
> we shouldn't be hitting the oom. We might have somebody freeing memory
> after the last attempt, though...
>
> [...]
>> Should be very easy to reproduce with a low mem VM (e.g. 192MB) under KVM
>> with ext4 and Fedora 25 and some memory load and updating the VM.
>>
>> Any further progress?
> The linux-next (resp. mmotm tree) has new tracepoints which should help
> to tell us more about what is going on here. Could you try to enable
> oom/reclaim_retry_zone and vmscan/mm_vmscan_direct_reclaim_{begin,end}

Is this available in this version?

https://koji.fedoraproject.org/koji/buildinfo?buildID=862775

kernel-4.11.0-0.rc0.git5.1.fc26

How to enable?


Thnx.

Ciao,

gerhard

[toc] | [prev] | [next] | [standalone]

#1589272

On Tue 28-02-17 07:06:41, Gerhard Wiesinger wrote:
> On 27.02.2017 09:27, Michal Hocko wrote:
> >On Sun 26-02-17 09:40:42, Gerhard Wiesinger wrote:
> >>On 04.01.2017 10:11, Michal Hocko wrote:
> >>>>The VM stops working (e.g. not pingable) after around 8h (will be restarted
> >>>>automatically), happened serveral times.
> >>>>
> >>>>Had also further OOMs which I sent to Mincham.
> >>>Could you post them to the mailing list as well, please?
> >>Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as
> >>well on 4.9.9-200.fc25.x86_64
> >>
> >>On 4.10er kernels:
> >[...]
> >>kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB
> >>active_anon:143580kB inactive_anon:143300kB active_file:2576kB
> >>inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB
> >>managed:353968kB mlocked:0kB slab_reclaimable:13708kB
> >>slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB
> >>free_pcp:412kB local_pcp:88kB free_cma:0kB
> >[...]
> >
> >>On 4.9er kernels:
> >[...]
> >>kernel: Node 0 DMA32 free:3356kB min:2668kB low:3332kB high:3996kB
> >>active_anon:122148kB inactive_anon:112068kB active_file:81324kB
> >>inactive_file:101972kB unevictable:0kB writepending:4648kB present:507760kB
> >>managed:484384kB mlocked:0kB slab_reclaimable:17660kB
> >>slab_unreclaimable:21404kB kernel_stack:2432kB pagetables:10124kB bounce:0kB
> >>free_pcp:120kB local_pcp:0kB free_cma:0kB
> >In both cases the amount if free memory is above the min watermark, so
> >we shouldn't be hitting the oom. We might have somebody freeing memory
> >after the last attempt, though...
> >
> >[...]
> >>Should be very easy to reproduce with a low mem VM (e.g. 192MB) under KVM
> >>with ext4 and Fedora 25 and some memory load and updating the VM.
> >>
> >>Any further progress?
> >The linux-next (resp. mmotm tree) has new tracepoints which should help
> >to tell us more about what is going on here. Could you try to enable
> >oom/reclaim_retry_zone and vmscan/mm_vmscan_direct_reclaim_{begin,end}
> 
> Is this available in this version?
> 
> https://koji.fedoraproject.org/koji/buildinfo?buildID=862775
> 
> kernel-4.11.0-0.rc0.git5.1.fc26

no idea.

> 
> How to enable?

mount -t tracefs none /trace
echo 1 > /trace/events/oom/reclaim_retry_zone/enabled
echo 1 > /trace/events/vmscan/mm_vmscan_direct_reclaim_begin
echo 1 > /trace/events/vmscan/mm_vmscan_direct_reclaim_end
-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1588559

On Sun, Feb 26, 2017 at 09:40:42AM +0100, Gerhard Wiesinger wrote:
> On 04.01.2017 10:11, Michal Hocko wrote:
> >>The VM stops working (e.g. not pingable) after around 8h (will be restarted
> >>automatically), happened serveral times.
> >>
> >>Had also further OOMs which I sent to Mincham.
> >Could you post them to the mailing list as well, please?
> 
> Still OOMs on dnf update procedure with kernel 4.10: 4.10.0-1.fc26.x86_64 as
> well on 4.9.9-200.fc25.x86_64
> 
> On 4.10er kernels:
> 
> Free swap  = 1137532kB
> 
> cat /etc/sysctl.d/* | grep ^vm
> vm.dirty_background_ratio = 3
> vm.dirty_ratio = 15
> vm.overcommit_memory = 2
> vm.overcommit_ratio = 80
> vm.swappiness=10
> 
> kernel: python invoked oom-killer:
> gfp_mask=0x14201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0,
> oom_score_adj=0
> kernel: python cpuset=/ mems_allowed=0
> kernel: CPU: 1 PID: 813 Comm: python Not tainted 4.10.0-1.fc26.x86_64 #1
> kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.3
> 04/01/2014
> kernel: Call Trace:
> kernel:  dump_stack+0x63/0x84
> kernel:  dump_header+0x7b/0x1f6
> kernel:  ? do_try_to_free_pages+0x2c5/0x340
> kernel:  oom_kill_process+0x202/0x3d0
> kernel:  out_of_memory+0x2b7/0x4e0
> kernel:  __alloc_pages_slowpath+0x915/0xb80
> kernel:  __alloc_pages_nodemask+0x218/0x2d0
> kernel:  alloc_pages_current+0x93/0x150
> kernel:  __page_cache_alloc+0xcf/0x100
> kernel:  filemap_fault+0x39d/0x800
> kernel:  ? page_add_file_rmap+0xe5/0x200
> kernel:  ? filemap_map_pages+0x2e1/0x4e0
> kernel:  ext4_filemap_fault+0x36/0x50
> kernel:  __do_fault+0x21/0x110
> kernel:  handle_mm_fault+0xdd1/0x1410
> kernel:  ? swake_up+0x42/0x50
> kernel:  __do_page_fault+0x23f/0x4c0
> kernel:  trace_do_page_fault+0x41/0x120
> kernel:  do_async_page_fault+0x51/0xa0
> kernel:  async_page_fault+0x28/0x30
> kernel: RIP: 0033:0x7f0681ad6350
> kernel: RSP: 002b:00007ffcbdd238d8 EFLAGS: 00010246
> kernel: RAX: 00007f0681b0f960 RBX: 0000000000000000 RCX: 7fffffffffffffff
> kernel: RDX: 0000000000000000 RSI: 3ff0000000000000 RDI: 3ff0000000000000
> kernel: RBP: 00007f067461ab40 R08: 0000000000000000 R09: 3ff0000000000000
> kernel: R10: 0000556f1c6d8a80 R11: 0000000000000001 R12: 00007f0676d1a8d0
> kernel: R13: 0000000000000000 R14: 00007f06746168bc R15: 00007f0674385910
> kernel: Mem-Info:
> kernel: active_anon:37423 inactive_anon:37512 isolated_anon:0
>          active_file:462 inactive_file:603 isolated_file:0
>          unevictable:0 dirty:0 writeback:0 unstable:0
>          slab_reclaimable:3538 slab_unreclaimable:4818
>          mapped:859 shmem:9 pagetables:3370 bounce:0
>          free:1650 free_pcp:103 free_cma:0
> kernel: Node 0 active_anon:149380kB inactive_anon:149704kB
> active_file:1848kB inactive_file:3660kB unevictable:0kB isolated(anon):128kB
> isolated(file):0kB mapped:4580kB dirty:0kB writeback:380kB shmem:0kB
> shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 36kB writeback_tmp:0kB
> unstable:0kB pages_scanned:352 all_unreclaimable? no
> kernel: Node 0 DMA free:1484kB min:104kB low:128kB high:152kB
> active_anon:5660kB inactive_anon:6156kB active_file:56kB inactive_file:64kB
> unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB
> slab_reclaimable:444kB slab_unreclaimable:1208kB kernel_stack:32kB
> pagetables:592kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
> kernel: lowmem_reserve[]: 0 327 327 327 327
> kernel: Node 0 DMA32 free:5012kB min:2264kB low:2828kB high:3392kB
> active_anon:143580kB inactive_anon:143300kB active_file:2576kB
> inactive_file:2560kB unevictable:0kB writepending:0kB present:376688kB
> managed:353968kB mlocked:0kB slab_reclaimable:13708kB
> slab_unreclaimable:18064kB kernel_stack:2352kB pagetables:12888kB bounce:0kB
> free_pcp:412kB local_pcp:88kB free_cma:0kB
> kernel: lowmem_reserve[]: 0 0 0 0 0
> kernel: Node 0 DMA: 70*4kB (UMEH) 20*8kB (UMEH) 13*16kB (MH) 5*32kB (H)
> 4*64kB (H) 2*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB =
> 1576kB
> kernel: Node 0 DMA32: 1134*4kB (UMEH) 25*8kB (UMEH) 13*16kB (MH) 7*32kB (H)
> 3*64kB (H) 0*128kB 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5616kB
 
Althogh DMA32 zone has enough free memory, free memory includes H pageblock
which is reserved memory for high-order atomic allocation. That might be
a reason you cannot succeed watermark check for the allocation.

I tried to solve the issue in 4.9 time to use up the reserved memory before
the OOM and merged into 4.10 but I think there is a hole so could you apply
this patch on top of your 4.10? (To be clear, cannot apply it to 4.9)

From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001
From: Minchan Kim <minchan@kernel.org>
Date: Mon, 27 Feb 2017 17:39:06 +0900
Subject: [PATCH] mm: use up highatomic before OOM kill

Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org>
---
 mm/page_alloc.c | 14 ++++----------
 1 file changed, 4 insertions(+), 10 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 614cd0397ce3..e073cca4969e 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
 		*no_progress_loops = 0;
 	else
 		(*no_progress_loops)++;
-
-	/*
-	 * Make sure we converge to OOM if we cannot make any progress
-	 * several times in the row.
-	 */
-	if (*no_progress_loops > MAX_RECLAIM_RETRIES) {
-		/* Before OOM, exhaust highatomic_reserve */
-		return unreserve_highatomic_pageblock(ac, true);
-	}
-
 	/*
 	 * Keep reclaiming pages while there is a chance this will lead
 	 * somewhere.  If none of the target zones can satisfy our allocation
@@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	if (read_mems_allowed_retry(cpuset_mems_cookie))
 		goto retry_cpuset;
 
+	/* Before OOM, exhaust highatomic_reserve */
+	if (unreserve_highatomic_pageblock(ac, true))
+		goto retry;
+
 	/* Reclaim has failed us, start killing things */
 	page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress);
 	if (page)
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1588575

On Mon 27-02-17 18:02:36, Minchan Kim wrote:
[...]
> >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001
> From: Minchan Kim <minchan@kernel.org>
> Date: Mon, 27 Feb 2017 17:39:06 +0900
> Subject: [PATCH] mm: use up highatomic before OOM kill
> 
> Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org>
> ---
>  mm/page_alloc.c | 14 ++++----------
>  1 file changed, 4 insertions(+), 10 deletions(-)
> 
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 614cd0397ce3..e073cca4969e 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
>  		*no_progress_loops = 0;
>  	else
>  		(*no_progress_loops)++;
> -
> -	/*
> -	 * Make sure we converge to OOM if we cannot make any progress
> -	 * several times in the row.
> -	 */
> -	if (*no_progress_loops > MAX_RECLAIM_RETRIES) {
> -		/* Before OOM, exhaust highatomic_reserve */
> -		return unreserve_highatomic_pageblock(ac, true);
> -	}
> -
>  	/*
>  	 * Keep reclaiming pages while there is a chance this will lead
>  	 * somewhere.  If none of the target zones can satisfy our allocation
> @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
>  	if (read_mems_allowed_retry(cpuset_mems_cookie))
>  		goto retry_cpuset;
>  
> +	/* Before OOM, exhaust highatomic_reserve */
> +	if (unreserve_highatomic_pageblock(ac, true))
> +		goto retry;
> +

OK, this can help for higher order requests when we do not exhaust all
the retries and fail on compaction but I fail to see how can this help
for order-0 requets which was what happened in this case. I am not
saying this is wrong, though.

>  	/* Reclaim has failed us, start killing things */
>  	page = __alloc_pages_may_oom(gfp_mask, order, ac, &did_some_progress);
>  	if (page)
> -- 
> 2.7.4

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1589193

On Mon, Feb 27, 2017 at 10:44:49AM +0100, Michal Hocko wrote:
> On Mon 27-02-17 18:02:36, Minchan Kim wrote:
> [...]
> > >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001
> > From: Minchan Kim <minchan@kernel.org>
> > Date: Mon, 27 Feb 2017 17:39:06 +0900
> > Subject: [PATCH] mm: use up highatomic before OOM kill
> > 
> > Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org>
> > ---
> >  mm/page_alloc.c | 14 ++++----------
> >  1 file changed, 4 insertions(+), 10 deletions(-)
> > 
> > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > index 614cd0397ce3..e073cca4969e 100644
> > --- a/mm/page_alloc.c
> > +++ b/mm/page_alloc.c
> > @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
> >  		*no_progress_loops = 0;
> >  	else
> >  		(*no_progress_loops)++;
> > -
> > -	/*
> > -	 * Make sure we converge to OOM if we cannot make any progress
> > -	 * several times in the row.
> > -	 */
> > -	if (*no_progress_loops > MAX_RECLAIM_RETRIES) {
> > -		/* Before OOM, exhaust highatomic_reserve */
> > -		return unreserve_highatomic_pageblock(ac, true);
> > -	}
> > -
> >  	/*
> >  	 * Keep reclaiming pages while there is a chance this will lead
> >  	 * somewhere.  If none of the target zones can satisfy our allocation
> > @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
> >  	if (read_mems_allowed_retry(cpuset_mems_cookie))
> >  		goto retry_cpuset;
> >  
> > +	/* Before OOM, exhaust highatomic_reserve */
> > +	if (unreserve_highatomic_pageblock(ac, true))
> > +		goto retry;
> > +
> 
> OK, this can help for higher order requests when we do not exhaust all
> the retries and fail on compaction but I fail to see how can this help
> for order-0 requets which was what happened in this case. I am not
> saying this is wrong, though.

The should_reclaim_retry can return false although no_progress_loop is less
than MAX_RECLAIM_RETRIES unless eligible zones has enough reclaimable pages
by the progress_loop. In that case, unreserve_highatomic_pageblock cannot
be called so that VM can keep a pageblock(e.g., 2M) for highatomic reserve.
Then, zone_watermark_ok subtracts nr_reserved_highatomic pages for the
pass/fail decision whichs is very conservative but no choice for the hot
path performance. With that, order-0 allocation can be failed.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1589275

On Tue 28-02-17 14:17:23, Minchan Kim wrote:
> On Mon, Feb 27, 2017 at 10:44:49AM +0100, Michal Hocko wrote:
> > On Mon 27-02-17 18:02:36, Minchan Kim wrote:
> > [...]
> > > >From 9779a1c5d32e2edb64da5cdfcd6f9737b94a247a Mon Sep 17 00:00:00 2001
> > > From: Minchan Kim <minchan@kernel.org>
> > > Date: Mon, 27 Feb 2017 17:39:06 +0900
> > > Subject: [PATCH] mm: use up highatomic before OOM kill
> > > 
> > > Not-Yet-Signed-off-by: Minchan Kim <minchan@kernel.org>
> > > ---
> > >  mm/page_alloc.c | 14 ++++----------
> > >  1 file changed, 4 insertions(+), 10 deletions(-)
> > > 
> > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> > > index 614cd0397ce3..e073cca4969e 100644
> > > --- a/mm/page_alloc.c
> > > +++ b/mm/page_alloc.c
> > > @@ -3549,16 +3549,6 @@ should_reclaim_retry(gfp_t gfp_mask, unsigned order,
> > >  		*no_progress_loops = 0;
> > >  	else
> > >  		(*no_progress_loops)++;
> > > -
> > > -	/*
> > > -	 * Make sure we converge to OOM if we cannot make any progress
> > > -	 * several times in the row.
> > > -	 */
> > > -	if (*no_progress_loops > MAX_RECLAIM_RETRIES) {
> > > -		/* Before OOM, exhaust highatomic_reserve */
> > > -		return unreserve_highatomic_pageblock(ac, true);
> > > -	}
> > > -
> > >  	/*
> > >  	 * Keep reclaiming pages while there is a chance this will lead
> > >  	 * somewhere.  If none of the target zones can satisfy our allocation
> > > @@ -3821,6 +3811,10 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
> > >  	if (read_mems_allowed_retry(cpuset_mems_cookie))
> > >  		goto retry_cpuset;
> > >  
> > > +	/* Before OOM, exhaust highatomic_reserve */
> > > +	if (unreserve_highatomic_pageblock(ac, true))
> > > +		goto retry;
> > > +
> > 
> > OK, this can help for higher order requests when we do not exhaust all
> > the retries and fail on compaction but I fail to see how can this help
> > for order-0 requets which was what happened in this case. I am not
> > saying this is wrong, though.
> 
> The should_reclaim_retry can return false although no_progress_loop is less
> than MAX_RECLAIM_RETRIES unless eligible zones has enough reclaimable pages
> by the progress_loop.

Yes, sorry I should have been more clear. I was talking about this
particular case where we had a lot of reclaimable pages (a lot of
anonymous with the swap available).

-- 
Michal Hocko
SUSE Labs

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web