Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1257675 > unrolled thread

[lkp] [mm, page_alloc] 43993977ba: +88% OOM possibility

Started bykernel test robot <ying.huang@linux.intel.com>
First post2015-10-28 06:40 +0100
Last post2015-11-02 11:30 +0100
Articles 7 — 3 participants

Back to article view | Back to linux.kernel

Contents

  [lkp] [mm, page_alloc] 43993977ba: +88% OOM possibility kernel test robot <ying.huang@linux.intel.com> - 2015-10-28 06:40 +0100
    Re: [lkp] [mm, page_alloc] 43993977ba: +88% OOM possibility Michal Hocko <mhocko@kernel.org> - 2015-10-29 15:40 +0100
      Re: [lkp] [mm, page_alloc] 43993977ba: +88% OOM possibility Michal Hocko <mhocko@kernel.org> - 2015-10-30 11:40 +0100
        Re: [lkp] [mm, page_alloc] 43993977ba: +88% OOM possibility "Huang\, Ying" <ying.huang@linux.intel.com> - 2015-11-02 00:30 +0100
          Re: [lkp] [mm, page_alloc] 43993977ba: +88% OOM possibility Michal Hocko <mhocko@kernel.org> - 2015-11-02 08:50 +0100
            Re: [lkp] [mm, page_alloc] 43993977ba: +88% OOM possibility "Huang\, Ying" <ying.huang@linux.intel.com> - 2015-11-02 10:00 +0100
              Re: [lkp] [mm, page_alloc] 43993977ba: +88% OOM possibility Michal Hocko <mhocko@kernel.org> - 2015-11-02 11:30 +0100

#1257675 — [lkp] [mm, page_alloc] 43993977ba: +88% OOM possibility

Fromkernel test robot <ying.huang@linux.intel.com>
Date2015-10-28 06:40 +0100
Subject[lkp] [mm, page_alloc] 43993977ba: +88% OOM possibility
Message-ID<qorCO-6TG-9@gated-at.bofh.it>

[Multipart message — attachments visible in raw view] — view raw

FYI, we noticed the below changes on

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
commit 43993977baecd838d66ccabc7f682342fc6ff635 ("mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd")

We found the OOM possibility increased 88% in a virtual machine with 1G memory.

=========================================================================================
tbox_group/testcase/rootfs/kconfig/compiler/disk/fs/test:
  vm-kbuild-1G/xfstests/debian-x86_64-2015-02-07.cgz/x86_64-allyesdebian/gcc-4.9/4HDD/btrfs/generic-mid

commit: 
  74fad8a3a917b9e0a407af8a4150c61f7b836591
  43993977baecd838d66ccabc7f682342fc6ff635

74fad8a3a917b9e0 43993977baecd838d66ccabc7f 
---------------- -------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          1:24          -4%            :24    xfstests.generic.192.fail
          1:24          -4%            :24    xfstests.nr_fail
           :24          88%          21:24    dmesg.Mem-Info
           :24          62%          15:24    dmesg.page_allocation_failure:order:#,mode
           :24          88%          21:24    dmesg.warn_alloc_failed+0x
           :24          75%          18:24    last_state.is_incomplete_run
          1:24          -4%            :24    last_state.xfstests.exit_code.1
           :24          54%          13:24    last_state.xfstests.exit_code.143
           :24          71%          17:24    kmsg.SLAB:Unable_to_allocate_memory_on_node#(gfp=#)
          1:24          -4%            :24    kmsg.TDH<#>
          1:24          -4%            :24    kmsg.TDH<c7>
          1:24          -4%            :24    kmsg.TDT<#>
          1:24          -4%            :24    kmsg.TDT<c7>
          1:24          -4%            :24    kmsg.Tx_Queue<#>
          1:24          -4%            :24    kmsg.buffer_info[next_to_clean]
          1:24          -4%            :24    kmsg.e1000#:#:#eth0:Detected_Tx_Unit_Hang
          1:24          -4%            :24    kmsg.jiffies<#>
          1:24          -4%            :24    kmsg.jiffies<#c5c>
          1:24          -4%            :24    kmsg.next_to_clean<#>
          1:24          -4%            :24    kmsg.next_to_clean<c7>
          1:24          -4%            :24    kmsg.next_to_use<#>
          1:24          -4%            :24    kmsg.next_to_use<c9>
          1:24          -4%            :24    kmsg.next_to_watch.status<#>
          1:24          -4%            :24    kmsg.next_to_watch<#>
          1:24          -4%            :24    kmsg.next_to_watch<c8>
          1:24          -4%            :24    kmsg.time_stamp<#>
          1:24          -4%            :24    kmsg.time_stamp<#afd>


vm-kbuild-1G: qemu-system-x86_64 -enable-kvm -cpu Haswell,+smep,+smap
Memory: 1G


To reproduce:

        git clone git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml


Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Ying Huang

[toc] | [next] | [standalone]

#1258834

FromMichal Hocko <mhocko@kernel.org>
Date2015-10-29 15:40 +0100
Message-ID<qoWwW-1lC-21@gated-at.bofh.it>
In reply to#1257675
On Wed 28-10-15 13:36:02, kernel test robot wrote:
> FYI, we noticed the below changes on
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
> commit 43993977baecd838d66ccabc7f682342fc6ff635 ("mm, page_alloc: distinguish between being unable to sleep, unwilling to sleep and avoiding waking kswapd")
> 
> We found the OOM possibility increased 88% in a virtual machine with 1G memory.

Could you provide dmesg output from this test?

Thanks!
-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1259385

FromMichal Hocko <mhocko@kernel.org>
Date2015-10-30 11:40 +0100
Message-ID<qpfge-4FN-19@gated-at.bofh.it>
In reply to#1258834
On Fri 30-10-15 16:21:40, Huang, Ying wrote:
> Michal Hocko <mhocko@kernel.org> writes:
> 
> > On Wed 28-10-15 13:36:02, kernel test robot wrote:
> >> FYI, we noticed the below changes on
> >> 
> >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
> >> commit 43993977baecd838d66ccabc7f682342fc6ff635 ("mm, page_alloc:
> >> distinguish between being unable to sleep, unwilling to sleep and
> >> avoiding waking kswapd")
> >> 
> >> We found the OOM possibility increased 88% in a virtual machine with 1G memory.
> >
> > Could you provide dmesg output from this test?
> 
> Sure, Attached.

I can only see a single allocation failure warning:
kworker/u4:1: page allocation failure: order:0, mode:0x2204000

This is obviously a non sleeping allocation with ___GFP_KSWAPD_RECLAIM
set. ___GFP_HIGH (aka access to memory reserves) is not required so a
failure of such an allocation is something to be expected.

[ 2294.616369] Workqueue: btrfs-submit btrfs_submit_helper
[ 2294.616369]  0000000000000000 ffff88000d38f5e0 ffffffff8173f84c 0000000000000000
[ 2294.616369]  ffff88000d38f678 ffffffff811abaee 00000000ffffffff 000000010038f618
[ 2294.616369]  ffff8800584e4148 00000000ffffffff ffff8800584e2f00 0000000000000001
[ 2294.616369] Call Trace:
[ 2294.616369]  [<ffffffff8173f84c>] dump_stack+0x4b/0x63
[ 2294.616369]  [<ffffffff811abaee>] warn_alloc_failed+0x125/0x13d
[ 2294.616369]  [<ffffffff811aecce>] __alloc_pages_nodemask+0x7c9/0x915
[ 2294.616369]  [<ffffffff811ecc7b>] kmem_getpages+0x91/0x155
[ 2294.616369]  [<ffffffff811eef0d>] fallback_alloc+0x1cc/0x24c
[ 2294.616369]  [<ffffffff811eed32>] ____cache_alloc_node+0x151/0x160
[ 2294.616369]  [<ffffffff811ef1ed>] __kmalloc+0xb0/0x134
[ 2294.616369]  [<ffffffff8105d7a5>] ? sched_clock+0x9/0xb
[ 2294.616369]  [<ffffffff8187d929>] ? virtqueue_add+0x78/0x37f
[ 2294.616369]  [<ffffffff8187d929>] virtqueue_add+0x78/0x37f
[ 2294.616369]  [<ffffffff81114f72>] ? __lock_acquire+0x751/0xf55
[ 2294.616369]  [<ffffffff8187dca6>] virtqueue_add_sgs+0x76/0x85

The patch you are referring shouldn't make any change in this path
because alloc_indirect which I expect is the allocation failing here
does:
gfp &= ~(__GFP_HIGHMEM | __GFP_HIGH)

and that came in via b92b1b89a33c ("virtio: force vring descriptors to
be allocated from lowmem").

Are there more failed allocations during the test? The subject would
suggest so.

Thanks!

-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1260260

From"Huang\, Ying" <ying.huang@linux.intel.com>
Date2015-11-02 00:30 +0100
Message-ID<qqaeu-6c1-15@gated-at.bofh.it>
In reply to#1259385
Michal Hocko <mhocko@kernel.org> writes:

> On Fri 30-10-15 16:21:40, Huang, Ying wrote:
>> Michal Hocko <mhocko@kernel.org> writes:
>> 
>> > On Wed 28-10-15 13:36:02, kernel test robot wrote:
>> >> FYI, we noticed the below changes on
>> >> 
>> >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
>> >> commit 43993977baecd838d66ccabc7f682342fc6ff635 ("mm, page_alloc:
>> >> distinguish between being unable to sleep, unwilling to sleep and
>> >> avoiding waking kswapd")
>> >> 
>> >> We found the OOM possibility increased 88% in a virtual machine with 1G memory.
>> >
>> > Could you provide dmesg output from this test?
>> 
>> Sure, Attached.
>
> I can only see a single allocation failure warning:
> kworker/u4:1: page allocation failure: order:0, mode:0x2204000
>
> This is obviously a non sleeping allocation with ___GFP_KSWAPD_RECLAIM
> set. ___GFP_HIGH (aka access to memory reserves) is not required so a
> failure of such an allocation is something to be expected.
>
> [ 2294.616369] Workqueue: btrfs-submit btrfs_submit_helper
> [ 2294.616369]  0000000000000000 ffff88000d38f5e0 ffffffff8173f84c 0000000000000000
> [ 2294.616369]  ffff88000d38f678 ffffffff811abaee 00000000ffffffff 000000010038f618
> [ 2294.616369]  ffff8800584e4148 00000000ffffffff ffff8800584e2f00 0000000000000001
> [ 2294.616369] Call Trace:
> [ 2294.616369]  [<ffffffff8173f84c>] dump_stack+0x4b/0x63
> [ 2294.616369]  [<ffffffff811abaee>] warn_alloc_failed+0x125/0x13d
> [ 2294.616369]  [<ffffffff811aecce>] __alloc_pages_nodemask+0x7c9/0x915
> [ 2294.616369]  [<ffffffff811ecc7b>] kmem_getpages+0x91/0x155
> [ 2294.616369]  [<ffffffff811eef0d>] fallback_alloc+0x1cc/0x24c
> [ 2294.616369]  [<ffffffff811eed32>] ____cache_alloc_node+0x151/0x160
> [ 2294.616369]  [<ffffffff811ef1ed>] __kmalloc+0xb0/0x134
> [ 2294.616369]  [<ffffffff8105d7a5>] ? sched_clock+0x9/0xb
> [ 2294.616369]  [<ffffffff8187d929>] ? virtqueue_add+0x78/0x37f
> [ 2294.616369]  [<ffffffff8187d929>] virtqueue_add+0x78/0x37f
> [ 2294.616369]  [<ffffffff81114f72>] ? __lock_acquire+0x751/0xf55
> [ 2294.616369]  [<ffffffff8187dca6>] virtqueue_add_sgs+0x76/0x85
>
> The patch you are referring shouldn't make any change in this path
> because alloc_indirect which I expect is the allocation failing here
> does:
> gfp &= ~(__GFP_HIGHMEM | __GFP_HIGH)
>
> and that came in via b92b1b89a33c ("virtio: force vring descriptors to
> be allocated from lowmem").
>
> Are there more failed allocations during the test? The subject would
> suggest so.

We done 24 tests for the commit and 24 tests for its parent.  There is
no OOM in any test for the parent commit, but there are OOM in 21 tests
for this commit.  This is what I want to say in the subject.  Sorry for
confusing.

Best Regards,
Huang, Ying
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1260430

FromMichal Hocko <mhocko@kernel.org>
Date2015-11-02 08:50 +0100
Message-ID<qqi2l-2yF-3@gated-at.bofh.it>
In reply to#1260260
On Mon 02-11-15 07:20:37, Huang, Ying wrote:
> Michal Hocko <mhocko@kernel.org> writes:
> 
> > On Fri 30-10-15 16:21:40, Huang, Ying wrote:
> >> Michal Hocko <mhocko@kernel.org> writes:
> >> 
> >> > On Wed 28-10-15 13:36:02, kernel test robot wrote:
> >> >> FYI, we noticed the below changes on
> >> >> 
> >> >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
> >> >> commit 43993977baecd838d66ccabc7f682342fc6ff635 ("mm, page_alloc:
> >> >> distinguish between being unable to sleep, unwilling to sleep and
> >> >> avoiding waking kswapd")
> >> >> 
> >> >> We found the OOM possibility increased 88% in a virtual machine with 1G memory.
> >> >
> >> > Could you provide dmesg output from this test?
> >> 
> >> Sure, Attached.
> >
> > I can only see a single allocation failure warning:
> > kworker/u4:1: page allocation failure: order:0, mode:0x2204000
> >
> > This is obviously a non sleeping allocation with ___GFP_KSWAPD_RECLAIM
> > set. ___GFP_HIGH (aka access to memory reserves) is not required so a
> > failure of such an allocation is something to be expected.
> >
> > [ 2294.616369] Workqueue: btrfs-submit btrfs_submit_helper
> > [ 2294.616369]  0000000000000000 ffff88000d38f5e0 ffffffff8173f84c 0000000000000000
> > [ 2294.616369]  ffff88000d38f678 ffffffff811abaee 00000000ffffffff 000000010038f618
> > [ 2294.616369]  ffff8800584e4148 00000000ffffffff ffff8800584e2f00 0000000000000001
> > [ 2294.616369] Call Trace:
> > [ 2294.616369]  [<ffffffff8173f84c>] dump_stack+0x4b/0x63
> > [ 2294.616369]  [<ffffffff811abaee>] warn_alloc_failed+0x125/0x13d
> > [ 2294.616369]  [<ffffffff811aecce>] __alloc_pages_nodemask+0x7c9/0x915
> > [ 2294.616369]  [<ffffffff811ecc7b>] kmem_getpages+0x91/0x155
> > [ 2294.616369]  [<ffffffff811eef0d>] fallback_alloc+0x1cc/0x24c
> > [ 2294.616369]  [<ffffffff811eed32>] ____cache_alloc_node+0x151/0x160
> > [ 2294.616369]  [<ffffffff811ef1ed>] __kmalloc+0xb0/0x134
> > [ 2294.616369]  [<ffffffff8105d7a5>] ? sched_clock+0x9/0xb
> > [ 2294.616369]  [<ffffffff8187d929>] ? virtqueue_add+0x78/0x37f
> > [ 2294.616369]  [<ffffffff8187d929>] virtqueue_add+0x78/0x37f
> > [ 2294.616369]  [<ffffffff81114f72>] ? __lock_acquire+0x751/0xf55
> > [ 2294.616369]  [<ffffffff8187dca6>] virtqueue_add_sgs+0x76/0x85
> >
> > The patch you are referring shouldn't make any change in this path
> > because alloc_indirect which I expect is the allocation failing here
> > does:
> > gfp &= ~(__GFP_HIGHMEM | __GFP_HIGH)
> >
> > and that came in via b92b1b89a33c ("virtio: force vring descriptors to
> > be allocated from lowmem").
> >
> > Are there more failed allocations during the test? The subject would
> > suggest so.
> 
> We done 24 tests for the commit and 24 tests for its parent.  There is
> no OOM in any test for the parent commit, but there are OOM in 21 tests
> for this commit.  This is what I want to say in the subject.  Sorry for
> confusing.

It would be interesting to see all the page allocation failure warnings
(if they are different). Maybe other callers have relied on GFP_ATOMIC
and access to memory reserves. The above path is not this case though.
-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1260458

From"Huang\, Ying" <ying.huang@linux.intel.com>
Date2015-11-02 10:00 +0100
Message-ID<qqj86-3aX-5@gated-at.bofh.it>
In reply to#1260430
Michal Hocko <mhocko@kernel.org> writes:

> On Mon 02-11-15 07:20:37, Huang, Ying wrote:
>> Michal Hocko <mhocko@kernel.org> writes:
>> 
>> > On Fri 30-10-15 16:21:40, Huang, Ying wrote:
>> >> Michal Hocko <mhocko@kernel.org> writes:
>> >> 
>> >> > On Wed 28-10-15 13:36:02, kernel test robot wrote:
>> >> >> FYI, we noticed the below changes on
>> >> >> 
>> >> >> https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master
>> >> >> commit 43993977baecd838d66ccabc7f682342fc6ff635 ("mm, page_alloc:
>> >> >> distinguish between being unable to sleep, unwilling to sleep and
>> >> >> avoiding waking kswapd")
>> >> >> 
>> >> >> We found the OOM possibility increased 88% in a virtual machine with 1G memory.
>> >> >
>> >> > Could you provide dmesg output from this test?
>> >> 
>> >> Sure, Attached.
>> >
>> > I can only see a single allocation failure warning:
>> > kworker/u4:1: page allocation failure: order:0, mode:0x2204000
>> >
>> > This is obviously a non sleeping allocation with ___GFP_KSWAPD_RECLAIM
>> > set. ___GFP_HIGH (aka access to memory reserves) is not required so a
>> > failure of such an allocation is something to be expected.
>> >
>> > [ 2294.616369] Workqueue: btrfs-submit btrfs_submit_helper
>> > [ 2294.616369]  0000000000000000 ffff88000d38f5e0 ffffffff8173f84c 0000000000000000
>> > [ 2294.616369]  ffff88000d38f678 ffffffff811abaee 00000000ffffffff 000000010038f618
>> > [ 2294.616369]  ffff8800584e4148 00000000ffffffff ffff8800584e2f00 0000000000000001
>> > [ 2294.616369] Call Trace:
>> > [ 2294.616369]  [<ffffffff8173f84c>] dump_stack+0x4b/0x63
>> > [ 2294.616369]  [<ffffffff811abaee>] warn_alloc_failed+0x125/0x13d
>> > [ 2294.616369]  [<ffffffff811aecce>] __alloc_pages_nodemask+0x7c9/0x915
>> > [ 2294.616369]  [<ffffffff811ecc7b>] kmem_getpages+0x91/0x155
>> > [ 2294.616369]  [<ffffffff811eef0d>] fallback_alloc+0x1cc/0x24c
>> > [ 2294.616369]  [<ffffffff811eed32>] ____cache_alloc_node+0x151/0x160
>> > [ 2294.616369]  [<ffffffff811ef1ed>] __kmalloc+0xb0/0x134
>> > [ 2294.616369]  [<ffffffff8105d7a5>] ? sched_clock+0x9/0xb
>> > [ 2294.616369]  [<ffffffff8187d929>] ? virtqueue_add+0x78/0x37f
>> > [ 2294.616369]  [<ffffffff8187d929>] virtqueue_add+0x78/0x37f
>> > [ 2294.616369]  [<ffffffff81114f72>] ? __lock_acquire+0x751/0xf55
>> > [ 2294.616369]  [<ffffffff8187dca6>] virtqueue_add_sgs+0x76/0x85
>> >
>> > The patch you are referring shouldn't make any change in this path
>> > because alloc_indirect which I expect is the allocation failing here
>> > does:
>> > gfp &= ~(__GFP_HIGHMEM | __GFP_HIGH)
>> >
>> > and that came in via b92b1b89a33c ("virtio: force vring descriptors to
>> > be allocated from lowmem").
>> >
>> > Are there more failed allocations during the test? The subject would
>> > suggest so.
>> 
>> We done 24 tests for the commit and 24 tests for its parent.  There is
>> no OOM in any test for the parent commit, but there are OOM in 21 tests
>> for this commit.  This is what I want to say in the subject.  Sorry for
>> confusing.
>
> It would be interesting to see all the page allocation failure warnings
> (if they are different). Maybe other callers have relied on GFP_ATOMIC
> and access to memory reserves. The above path is not this case though.

I take a look at all dmesgs, and found the backtrace for page allocation
failure is same for all.  Is it possible that this commit cause more
memory were allocated or kept in memory so that more OOM were triggered?

Best Regards,
Huang, Ying
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1260520

FromMichal Hocko <mhocko@kernel.org>
Date2015-11-02 11:30 +0100
Message-ID<qqkxc-48H-23@gated-at.bofh.it>
In reply to#1260458
On Mon 02-11-15 16:55:15, Huang, Ying wrote:
> Michal Hocko <mhocko@kernel.org> writes:
[...]
> > It would be interesting to see all the page allocation failure warnings
> > (if they are different). Maybe other callers have relied on GFP_ATOMIC
> > and access to memory reserves. The above path is not this case though.
> 
> I take a look at all dmesgs, and found the backtrace for page allocation
> failure is same for all.  Is it possible that this commit cause more
> memory were allocated or kept in memory so that more OOM were triggered?

I can imagine that some of the callers were not converted properly or
missed and a lack of __GFP_KSWAPD_RECLAIM could indeed cause a later
kswapd kick off. I am staring into the commit but nothing has jumped at
me yet. Could you collect /proc/vmstat (snapshot every 1s) on both good
and bad kernels. I expect the later would see a less scanning by kswapd.
-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web