Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1281635 > unrolled thread

Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead

Started byMel Gorman <mgorman@techsingularity.net>
First post2015-12-02 12:10 +0100
Last post2015-12-07 17:20 +0100
Articles 9 — 3 participants

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Mel Gorman <mgorman@techsingularity.net> - 2015-12-02 12:10 +0100
    Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Michal Hocko <mhocko@kernel.org> - 2015-12-02 13:10 +0100
      Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Mel Gorman <mgorman@techsingularity.net> - 2015-12-02 15:10 +0100
        Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Michal Hocko <mhocko@kernel.org> - 2015-12-02 15:20 +0100
          Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Mel Gorman <mgorman@techsingularity.net> - 2015-12-02 15:50 +0100
            Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead "Huang\, Ying" <ying.huang@linux.intel.com> - 2015-12-03 09:50 +0100
              Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Mel Gorman <mgorman@techsingularity.net> - 2015-12-03 11:20 +0100
                Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead "Huang\, Ying" <ying.huang@linux.intel.com> - 2015-12-04 03:00 +0100
                  Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead Michal Hocko <mhocko@kernel.org> - 2015-12-07 17:20 +0100

#1281635 — Re: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead

FromMel Gorman <mgorman@techsingularity.net>
Date2015-12-02 12:10 +0100
SubjectRe: [lkp] [mm, page_alloc] d0164adc89: -100.0% fsmark.app_overhead
Message-ID<qBdsn-6oc-29@gated-at.bofh.it>
On Mon, Nov 30, 2015 at 10:14:24AM +0800, Huang, Ying wrote:
> > There is no reference to OOM possibility in the email that I can see. Can
> > you give examples of the OOM messages that shows the problem sites? It was
> > suspected that there may be some callers that were accidentally depending
> > on access to emergency reserves. If so, either they need to be fixed (if
> > the case is extremely rare) or a small reserve will have to be created
> > for callers that are not high priority but still cannot reclaim.
> >
> > Note that I'm travelling a lot over the next two weeks so I'll be slow to
> > respond but I will get to it.
> 
> Here is the kernel log,  the full dmesg is attached too.  The OOM
> occurs during fsmark testing.
> 
> Best Regards,
> Huang, Ying
> 
> [   31.453514] kworker/u4:0: page allocation failure: order:0, mode:0x2200000
> [   31.463570] CPU: 0 PID: 6 Comm: kworker/u4:0 Not tainted 4.3.0-08056-gd0164ad #1
> [   31.466115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014
> [   31.477146] Workqueue: writeback wb_workfn (flush-253:0)
> [   31.481450]  0000000000000000 ffff880035ac75e8 ffffffff8140a142 0000000002200000
> [   31.492582]  ffff880035ac7670 ffffffff8117117b ffff880037586b28 ffff880000000040
> [   31.507631]  ffff88003523b270 0000000000000040 ffff880035abc800 ffffffff00000000

This is an allocation failure and is not a triggering of the OOM killer so
the severity is reduced but it still looks like a bug in the driver. Looking
at the history and the discussion, it appears to me that __GFP_HIGH was
cleared from the allocation site by accident. I strongly suspect that Will
Deacon thought __GFP_HIGH was related to highmem instead of being related
to high priority.  Will, can you review the following patch please? Ying,
can you test please?

---8<---
virtio: allow vring descriptor allocations to use high-priority reserves

Commit b92b1b89a33c ("virtio: force vring descriptors to be allocated
from lowmem") prevented the inappropriate use of highmem pages but it
also masked out __GFP_HIGH. __GFP_HIGH is used for GFP_ATOMIC allocation
requests to grant access to a small emergency reserve. It's intended for
user by callers that have no alternative.

Ying Huang reported the following page allocation failure warning after
commit d0164adc89f6 ("mm, page_alloc: distinguish between being unable to
sleep, unwilling to sleep and avoiding waking kswapd")

    kworker/u4:0: page allocation failure: order:0, mode:0x2200000
    CPU: 0 PID: 6 Comm: kworker/u4:0 Not tainted 4.3.0-08056-gd0164ad #1
    Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014
    Workqueue: writeback wb_workfn (flush-253:0)
     0000000000000000 ffff880035ac75e8 ffffffff8140a142 0000000002200000
     ffff880035ac7670 ffffffff8117117b ffff880037586b28 ffff880000000040
     ffff88003523b270 0000000000000040 ffff880035abc800 ffffffff00000000
    Call Trace:
     [<ffffffff8140a142>] dump_stack+0x4b/0x69
     [<ffffffff8117117b>] warn_alloc_failed+0xdb/0x140
     [<ffffffff81174ec4>] __alloc_pages_nodemask+0x874/0xa60
     [<ffffffff811bcb62>] alloc_pages_current+0x92/0x120
     [<ffffffff811c73e4>] new_slab+0x3d4/0x480
     [<ffffffff811c7c36>] __slab_alloc+0x376/0x470
     [<ffffffff814e0ced>] ? alloc_indirect+0x1d/0x50
     [<ffffffff81338221>] ? xfs_submit_ioend_bio+0x31/0x40
     [<ffffffff814e0ced>] ? alloc_indirect+0x1d/0x50
     [<ffffffff811c8e8d>] __kmalloc+0x20d/0x260
     [<ffffffff814e0ced>] alloc_indirect+0x1d/0x50
     [<ffffffff814e0fec>] virtqueue_add_sgs+0x2cc/0x3a0
     [<ffffffff81573a30>] __virtblk_add_req+0xb0/0x1f0
     [<ffffffff8117a121>] ? pagevec_lookup_tag+0x21/0x30
     [<ffffffff813e5d72>] ? blk_rq_map_sg+0x1e2/0x4f0
     [<ffffffff81573c82>] virtio_queue_rq+0x112/0x280
     [<ffffffff813e9de7>] __blk_mq_run_hw_queue+0x1d7/0x370
     [<ffffffff813e9bef>] blk_mq_run_hw_queue+0x9f/0xc0
     [<ffffffff813eb10a>] blk_mq_insert_requests+0xfa/0x1a0
     [<ffffffff813ebdb3>] blk_mq_flush_plug_list+0x123/0x140
     [<ffffffff813e1777>] blk_flush_plug_list+0xa7/0x200
     [<ffffffff813e1c49>] blk_finish_plug+0x29/0x40
     [<ffffffff81215f85>] wb_writeback+0x185/0x2c0
     [<ffffffff812166a5>] wb_workfn+0xf5/0x390
     [<ffffffff81091297>] process_one_work+0x157/0x420
     [<ffffffff81091ef9>] worker_thread+0x69/0x4a0
     [<ffffffff81091e90>] ? rescuer_thread+0x380/0x380
     [<ffffffff8109746f>] kthread+0xef/0x110
     [<ffffffff81097380>] ? kthread_park+0x60/0x60
     [<ffffffff818bce8f>] ret_from_fork+0x3f/0x70
     [<ffffffff81097380>] ? kthread_park+0x60/0x60

Commit d0164adc89f6 ("mm, page_alloc: distinguish between being unable to
sleep, unwilling to sleep and avoiding waking kswapd") is stricter about
reserves. It distinguishes between callers that are high-priority with
access to emergency reserves and callers that simply do not want to sleep
and have recovery options. The reported allocation failure is truly atomic
with no recovery options that appears to have cleared __GFP_HIGH by mistake
for reasons that are unrelated to highmem. This patch restores the flag.

Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
 drivers/virtio/virtio_ring.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index 096b857e7b75..f9e119e6df18 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -107,9 +107,10 @@ static struct vring_desc *alloc_indirect(struct virtqueue *_vq,
 	/*
 	 * We require lowmem mappings for the descriptors because
 	 * otherwise virt_to_phys will give us bogus addresses in the
-	 * virtqueue.
+	 * virtqueue. Access to high-priority reserves is preserved
+	 * if originally requested by GFP_ATOMIC.
 	 */
-	gfp &= ~(__GFP_HIGHMEM | __GFP_HIGH);
+	gfp &= ~__GFP_HIGHMEM;
 
 	desc = kmalloc(total_sg * sizeof(struct vring_desc), gfp);
 	if (!desc)
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [next] | [standalone]

#1281704

FromMichal Hocko <mhocko@kernel.org>
Date2015-12-02 13:10 +0100
Message-ID<qBeoq-6Z9-3@gated-at.bofh.it>
In reply to#1281635
On Wed 02-12-15 11:00:09, Mel Gorman wrote:
> On Mon, Nov 30, 2015 at 10:14:24AM +0800, Huang, Ying wrote:
> > > There is no reference to OOM possibility in the email that I can see. Can
> > > you give examples of the OOM messages that shows the problem sites? It was
> > > suspected that there may be some callers that were accidentally depending
> > > on access to emergency reserves. If so, either they need to be fixed (if
> > > the case is extremely rare) or a small reserve will have to be created
> > > for callers that are not high priority but still cannot reclaim.
> > >
> > > Note that I'm travelling a lot over the next two weeks so I'll be slow to
> > > respond but I will get to it.
> > 
> > Here is the kernel log,  the full dmesg is attached too.  The OOM
> > occurs during fsmark testing.
> > 
> > Best Regards,
> > Huang, Ying
> > 
> > [   31.453514] kworker/u4:0: page allocation failure: order:0, mode:0x2200000
> > [   31.463570] CPU: 0 PID: 6 Comm: kworker/u4:0 Not tainted 4.3.0-08056-gd0164ad #1
> > [   31.466115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014
> > [   31.477146] Workqueue: writeback wb_workfn (flush-253:0)
> > [   31.481450]  0000000000000000 ffff880035ac75e8 ffffffff8140a142 0000000002200000
> > [   31.492582]  ffff880035ac7670 ffffffff8117117b ffff880037586b28 ffff880000000040
> > [   31.507631]  ffff88003523b270 0000000000000040 ffff880035abc800 ffffffff00000000
> 
> This is an allocation failure and is not a triggering of the OOM killer so
> the severity is reduced but it still looks like a bug in the driver. Looking
> at the history and the discussion, it appears to me that __GFP_HIGH was
> cleared from the allocation site by accident. I strongly suspect that Will
> Deacon thought __GFP_HIGH was related to highmem instead of being related
> to high priority.  Will, can you review the following patch please? Ying,
> can you test please?

I have posted basically the same patch
http://lkml.kernel.org/r/1448980369-27130-1-git-send-email-mhocko@kernel.org

I didn't mention this allocation failure because I am not sure it is
really related.

> ---8<---
> virtio: allow vring descriptor allocations to use high-priority reserves
> 
> Commit b92b1b89a33c ("virtio: force vring descriptors to be allocated
> from lowmem") prevented the inappropriate use of highmem pages but it
> also masked out __GFP_HIGH. __GFP_HIGH is used for GFP_ATOMIC allocation
> requests to grant access to a small emergency reserve. It's intended for
> user by callers that have no alternative.
> 
> Ying Huang reported the following page allocation failure warning after
> commit d0164adc89f6 ("mm, page_alloc: distinguish between being unable to
> sleep, unwilling to sleep and avoiding waking kswapd")
> 
>     kworker/u4:0: page allocation failure: order:0, mode:0x2200000
>     CPU: 0 PID: 6 Comm: kworker/u4:0 Not tainted 4.3.0-08056-gd0164ad #1
>     Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014
>     Workqueue: writeback wb_workfn (flush-253:0)
>      0000000000000000 ffff880035ac75e8 ffffffff8140a142 0000000002200000
>      ffff880035ac7670 ffffffff8117117b ffff880037586b28 ffff880000000040
>      ffff88003523b270 0000000000000040 ffff880035abc800 ffffffff00000000
>     Call Trace:
>      [<ffffffff8140a142>] dump_stack+0x4b/0x69
>      [<ffffffff8117117b>] warn_alloc_failed+0xdb/0x140
>      [<ffffffff81174ec4>] __alloc_pages_nodemask+0x874/0xa60
>      [<ffffffff811bcb62>] alloc_pages_current+0x92/0x120
>      [<ffffffff811c73e4>] new_slab+0x3d4/0x480
>      [<ffffffff811c7c36>] __slab_alloc+0x376/0x470
>      [<ffffffff814e0ced>] ? alloc_indirect+0x1d/0x50
>      [<ffffffff81338221>] ? xfs_submit_ioend_bio+0x31/0x40
>      [<ffffffff814e0ced>] ? alloc_indirect+0x1d/0x50
>      [<ffffffff811c8e8d>] __kmalloc+0x20d/0x260
>      [<ffffffff814e0ced>] alloc_indirect+0x1d/0x50
>      [<ffffffff814e0fec>] virtqueue_add_sgs+0x2cc/0x3a0
>      [<ffffffff81573a30>] __virtblk_add_req+0xb0/0x1f0
>      [<ffffffff8117a121>] ? pagevec_lookup_tag+0x21/0x30
>      [<ffffffff813e5d72>] ? blk_rq_map_sg+0x1e2/0x4f0
>      [<ffffffff81573c82>] virtio_queue_rq+0x112/0x280
>      [<ffffffff813e9de7>] __blk_mq_run_hw_queue+0x1d7/0x370
>      [<ffffffff813e9bef>] blk_mq_run_hw_queue+0x9f/0xc0
>      [<ffffffff813eb10a>] blk_mq_insert_requests+0xfa/0x1a0
>      [<ffffffff813ebdb3>] blk_mq_flush_plug_list+0x123/0x140
>      [<ffffffff813e1777>] blk_flush_plug_list+0xa7/0x200
>      [<ffffffff813e1c49>] blk_finish_plug+0x29/0x40
>      [<ffffffff81215f85>] wb_writeback+0x185/0x2c0
>      [<ffffffff812166a5>] wb_workfn+0xf5/0x390
>      [<ffffffff81091297>] process_one_work+0x157/0x420
>      [<ffffffff81091ef9>] worker_thread+0x69/0x4a0
>      [<ffffffff81091e90>] ? rescuer_thread+0x380/0x380
>      [<ffffffff8109746f>] kthread+0xef/0x110
>      [<ffffffff81097380>] ? kthread_park+0x60/0x60
>      [<ffffffff818bce8f>] ret_from_fork+0x3f/0x70
>      [<ffffffff81097380>] ? kthread_park+0x60/0x60
> 
> Commit d0164adc89f6 ("mm, page_alloc: distinguish between being unable to
> sleep, unwilling to sleep and avoiding waking kswapd") is stricter about
> reserves. It distinguishes between callers that are high-priority with
> access to emergency reserves and callers that simply do not want to sleep
> and have recovery options. The reported allocation failure is truly atomic
> with no recovery options that appears to have cleared __GFP_HIGH by mistake
> for reasons that are unrelated to highmem. This patch restores the flag.
> 
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
> ---
>  drivers/virtio/virtio_ring.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
> index 096b857e7b75..f9e119e6df18 100644
> --- a/drivers/virtio/virtio_ring.c
> +++ b/drivers/virtio/virtio_ring.c
> @@ -107,9 +107,10 @@ static struct vring_desc *alloc_indirect(struct virtqueue *_vq,
>  	/*
>  	 * We require lowmem mappings for the descriptors because
>  	 * otherwise virt_to_phys will give us bogus addresses in the
> -	 * virtqueue.
> +	 * virtqueue. Access to high-priority reserves is preserved
> +	 * if originally requested by GFP_ATOMIC.
>  	 */
> -	gfp &= ~(__GFP_HIGHMEM | __GFP_HIGH);
> +	gfp &= ~__GFP_HIGHMEM;
>  
>  	desc = kmalloc(total_sg * sizeof(struct vring_desc), gfp);
>  	if (!desc)

-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1281841

FromMel Gorman <mgorman@techsingularity.net>
Date2015-12-02 15:10 +0100
Message-ID<qBggB-8db-73@gated-at.bofh.it>
In reply to#1281704
On Wed, Dec 02, 2015 at 01:00:46PM +0100, Michal Hocko wrote:
> On Wed 02-12-15 11:00:09, Mel Gorman wrote:
> > On Mon, Nov 30, 2015 at 10:14:24AM +0800, Huang, Ying wrote:
> > > > There is no reference to OOM possibility in the email that I can see. Can
> > > > you give examples of the OOM messages that shows the problem sites? It was
> > > > suspected that there may be some callers that were accidentally depending
> > > > on access to emergency reserves. If so, either they need to be fixed (if
> > > > the case is extremely rare) or a small reserve will have to be created
> > > > for callers that are not high priority but still cannot reclaim.
> > > >
> > > > Note that I'm travelling a lot over the next two weeks so I'll be slow to
> > > > respond but I will get to it.
> > > 
> > > Here is the kernel log,  the full dmesg is attached too.  The OOM
> > > occurs during fsmark testing.
> > > 
> > > Best Regards,
> > > Huang, Ying
> > > 
> > > [   31.453514] kworker/u4:0: page allocation failure: order:0, mode:0x2200000
> > > [   31.463570] CPU: 0 PID: 6 Comm: kworker/u4:0 Not tainted 4.3.0-08056-gd0164ad #1
> > > [   31.466115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014
> > > [   31.477146] Workqueue: writeback wb_workfn (flush-253:0)
> > > [   31.481450]  0000000000000000 ffff880035ac75e8 ffffffff8140a142 0000000002200000
> > > [   31.492582]  ffff880035ac7670 ffffffff8117117b ffff880037586b28 ffff880000000040
> > > [   31.507631]  ffff88003523b270 0000000000000040 ffff880035abc800 ffffffff00000000
> > 
> > This is an allocation failure and is not a triggering of the OOM killer so
> > the severity is reduced but it still looks like a bug in the driver. Looking
> > at the history and the discussion, it appears to me that __GFP_HIGH was
> > cleared from the allocation site by accident. I strongly suspect that Will
> > Deacon thought __GFP_HIGH was related to highmem instead of being related
> > to high priority.  Will, can you review the following patch please? Ying,
> > can you test please?
> 
> I have posted basically the same patch
> http://lkml.kernel.org/r/1448980369-27130-1-git-send-email-mhocko@kernel.org
> 

Sorry. I missed that while playing catch-up and I wasn't on the cc. I'll
drop this patch now. Thanks for catching it.

> I didn't mention this allocation failure because I am not sure it is
> really related.
> 

I'm fairly sure it is. The failure is an allocation site that cannot
sleep but did not specify __GFP_HIGH. Such callers are normally expected
to be able to recover gracefully and probably should specify _GFP_NOWARN.
kswapd would have woken up as normal but the free pages were below the
min watermark so there was a brief failure.

-- 
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1281845

FromMichal Hocko <mhocko@kernel.org>
Date2015-12-02 15:20 +0100
Message-ID<qBgqd-8gM-5@gated-at.bofh.it>
In reply to#1281841
On Wed 02-12-15 14:08:52, Mel Gorman wrote:
> On Wed, Dec 02, 2015 at 01:00:46PM +0100, Michal Hocko wrote:
> > On Wed 02-12-15 11:00:09, Mel Gorman wrote:
> > > On Mon, Nov 30, 2015 at 10:14:24AM +0800, Huang, Ying wrote:
> > > > > There is no reference to OOM possibility in the email that I can see. Can
> > > > > you give examples of the OOM messages that shows the problem sites? It was
> > > > > suspected that there may be some callers that were accidentally depending
> > > > > on access to emergency reserves. If so, either they need to be fixed (if
> > > > > the case is extremely rare) or a small reserve will have to be created
> > > > > for callers that are not high priority but still cannot reclaim.
> > > > >
> > > > > Note that I'm travelling a lot over the next two weeks so I'll be slow to
> > > > > respond but I will get to it.
> > > > 
> > > > Here is the kernel log,  the full dmesg is attached too.  The OOM
> > > > occurs during fsmark testing.
> > > > 
> > > > Best Regards,
> > > > Huang, Ying
> > > > 
> > > > [   31.453514] kworker/u4:0: page allocation failure: order:0, mode:0x2200000
> > > > [   31.463570] CPU: 0 PID: 6 Comm: kworker/u4:0 Not tainted 4.3.0-08056-gd0164ad #1
> > > > [   31.466115] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014
> > > > [   31.477146] Workqueue: writeback wb_workfn (flush-253:0)
> > > > [   31.481450]  0000000000000000 ffff880035ac75e8 ffffffff8140a142 0000000002200000
> > > > [   31.492582]  ffff880035ac7670 ffffffff8117117b ffff880037586b28 ffff880000000040
> > > > [   31.507631]  ffff88003523b270 0000000000000040 ffff880035abc800 ffffffff00000000
> > > 
> > > This is an allocation failure and is not a triggering of the OOM killer so
> > > the severity is reduced but it still looks like a bug in the driver. Looking
> > > at the history and the discussion, it appears to me that __GFP_HIGH was
> > > cleared from the allocation site by accident. I strongly suspect that Will
> > > Deacon thought __GFP_HIGH was related to highmem instead of being related
> > > to high priority.  Will, can you review the following patch please? Ying,
> > > can you test please?
> > 
> > I have posted basically the same patch
> > http://lkml.kernel.org/r/1448980369-27130-1-git-send-email-mhocko@kernel.org
> > 
> 
> Sorry. I missed that while playing catch-up and I wasn't on the cc. I'll
> drop this patch now. Thanks for catching it.

My bad. I should have CCed you. But I considered this merely a cleanup
so I didn't want to swamp you with another email.

> > I didn't mention this allocation failure because I am not sure it is
> > really related.
> > 
> 
> I'm fairly sure it is. The failure is an allocation site that cannot
> sleep but did not specify __GFP_HIGH.

yeah but this was the case even before your patch. As the caller used
GFP_ATOMIC then it got __GFP_ATOMIC after your patch so it still
managed to do ALLOC_HARDER. I would agree if this was an explicit
GFP_NOWAIT. Unless I am missing something your patch hasn't changed the
behavior for this particular allocation.

> Such callers are normally expected
> to be able to recover gracefully and probably should specify _GFP_NOWARN.
> kswapd would have woken up as normal but the free pages were below the
> min watermark so there was a brief failure.

-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1281871

FromMel Gorman <mgorman@techsingularity.net>
Date2015-12-02 15:50 +0100
Message-ID<qBgTf-8qM-3@gated-at.bofh.it>
In reply to#1281845
On Wed, Dec 02, 2015 at 03:15:29PM +0100, Michal Hocko wrote:
> > > I didn't mention this allocation failure because I am not sure it is
> > > really related.
> > > 
> > 
> > I'm fairly sure it is. The failure is an allocation site that cannot
> > sleep but did not specify __GFP_HIGH.
> 
> yeah but this was the case even before your patch. As the caller used
> GFP_ATOMIC then it got __GFP_ATOMIC after your patch so it still
> managed to do ALLOC_HARDER. I would agree if this was an explicit
> GFP_NOWAIT. Unless I am missing something your patch hasn't changed the
> behavior for this particular allocation.
> 

You're right. I think it's this hunk that is the problem.

@@ -1186,7 +1186,7 @@ static struct request *blk_mq_map_request(struct
request_queue *q,
                ctx = blk_mq_get_ctx(q);
                hctx = q->mq_ops->map_queue(q, ctx->cpu);
                blk_mq_set_alloc_data(&alloc_data, q,
-                               __GFP_WAIT|GFP_ATOMIC, false, ctx, hctx);
+                               __GFP_WAIT|__GFP_HIGH, false, ctx, hctx);
                rq = __blk_mq_alloc_request(&alloc_data, rw);
                ctx = alloc_data.ctx;
                hctx = alloc_data.hctx;

This specific path at this patch is not waking kswapd any more when it
should. A series of allocations there could hit the watermarks and never wake
kswapd and then be followed by an atomic allocation failure that woke kswapd.

This bug gets fixed later by the commit 71baba4b92dc ("mm, page_alloc:
rename __GFP_WAIT to __GFP_RECLAIM") so it's not a bug in the current
kernel. However, it happens to break bisection and would be caught if each
individual commit was tested.

Your __GFP_HIGH patch is still fine although not the direct fix for this
specific problem. Commit 71baba4b92dc is.

Ying, does the page allocation failure messages happen when the whole
series is applied? i.e. is 4.4-rc3 ok?

-- 
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1282807

From"Huang\, Ying" <ying.huang@linux.intel.com>
Date2015-12-03 09:50 +0100
Message-ID<qBxKp-2zO-3@gated-at.bofh.it>
In reply to#1281871

[Multipart message — attachments visible in raw view] — view raw

Mel Gorman <mgorman@techsingularity.net> writes:

> On Wed, Dec 02, 2015 at 03:15:29PM +0100, Michal Hocko wrote:
>> > > I didn't mention this allocation failure because I am not sure it is
>> > > really related.
>> > > 
>> > 
>> > I'm fairly sure it is. The failure is an allocation site that cannot
>> > sleep but did not specify __GFP_HIGH.
>> 
>> yeah but this was the case even before your patch. As the caller used
>> GFP_ATOMIC then it got __GFP_ATOMIC after your patch so it still
>> managed to do ALLOC_HARDER. I would agree if this was an explicit
>> GFP_NOWAIT. Unless I am missing something your patch hasn't changed the
>> behavior for this particular allocation.
>> 
>
> You're right. I think it's this hunk that is the problem.
>
> @@ -1186,7 +1186,7 @@ static struct request *blk_mq_map_request(struct
> request_queue *q,
>                 ctx = blk_mq_get_ctx(q);
>                 hctx = q->mq_ops->map_queue(q, ctx->cpu);
>                 blk_mq_set_alloc_data(&alloc_data, q,
> -                               __GFP_WAIT|GFP_ATOMIC, false, ctx, hctx);
> +                               __GFP_WAIT|__GFP_HIGH, false, ctx, hctx);
>                 rq = __blk_mq_alloc_request(&alloc_data, rw);
>                 ctx = alloc_data.ctx;
>                 hctx = alloc_data.hctx;
>
> This specific path at this patch is not waking kswapd any more when it
> should. A series of allocations there could hit the watermarks and never wake
> kswapd and then be followed by an atomic allocation failure that woke kswapd.
>
> This bug gets fixed later by the commit 71baba4b92dc ("mm, page_alloc:
> rename __GFP_WAIT to __GFP_RECLAIM") so it's not a bug in the current
> kernel. However, it happens to break bisection and would be caught if each
> individual commit was tested.
>
> Your __GFP_HIGH patch is still fine although not the direct fix for this
> specific problem. Commit 71baba4b92dc is.
>
> Ying, does the page allocation failure messages happen when the whole
> series is applied? i.e. is 4.4-rc3 ok?

There are allocation errors for 4.4-rc3 too. dmesg is attached.

[   54.970840] kworker/u4:0: page allocation failure: order:0, mode:0x2204000
[   54.973161] CPU: 1 PID: 6 Comm: kworker/u4:0 Not tainted 4.4.0-rc3 #1
[   54.975179] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Debian-1.8.2-1 04/01/2014
[   54.980562] Workqueue: writeback wb_workfn (flush-253:0)
[   54.982289]  0000000000000000 ffff880035ac7098 ffffffff81410ad2 0000000002204000
[   54.985438]  ffff880035ac7120 ffffffff811724cb ffff880037586b50 0000000000000040
[   54.988746]  0000000000000000 0000000000000040 ffffffff00ac70e0 ffffffff00000000
[   54.991669] Call Trace:
[   54.992754]  [<ffffffff81410ad2>] dump_stack+0x4b/0x69
[   54.994433]  [<ffffffff811724cb>] warn_alloc_failed+0xdb/0x140
[   54.996134]  [<ffffffff811762b9>] __alloc_pages_nodemask+0x949/0xb40
[   54.997846]  [<ffffffff811be2c2>] alloc_pages_current+0x92/0x120
[   54.999494]  [<ffffffff811c8bfc>] new_slab+0x3dc/0x480
[   55.001007]  [<ffffffff811c9421>] ___slab_alloc+0x351/0x440
[   55.002588]  [<ffffffff814e7fcd>] ? alloc_indirect+0x1d/0x50
[   55.004268]  [<ffffffff8132270f>] ? xfs_btree_insrec+0x47f/0x700
[   55.005926]  [<ffffffff813206e9>] ? xfs_btree_delrec+0x5a9/0x1380
[   55.007592]  [<ffffffff814e7fcd>] ? alloc_indirect+0x1d/0x50
[   55.009281]  [<ffffffff811c9530>] __slab_alloc+0x20/0x40
[   55.010827]  [<ffffffff811ca03d>] __kmalloc+0x20d/0x260
[   55.012367]  [<ffffffff814e7fcd>] alloc_indirect+0x1d/0x50
[   55.014040]  [<ffffffff814e82cc>] virtqueue_add_sgs+0x2cc/0x3a0
[   55.015683]  [<ffffffff8157b240>] __virtblk_add_req+0xb0/0x1f0
[   55.017300]  [<ffffffff81307944>] ? xfs_alloc_update_counters+0x44/0x50
[   55.019759]  [<ffffffff813ec524>] ? blk_rq_map_sg+0x1f4/0x510
[   55.021371]  [<ffffffff8157b492>] virtio_queue_rq+0x112/0x280
[   55.022978]  [<ffffffff813f0647>] __blk_mq_run_hw_queue+0x1d7/0x370
[   55.024674]  [<ffffffff813f044f>] blk_mq_run_hw_queue+0x9f/0xc0
[   55.026307]  [<ffffffff813f196a>] blk_mq_insert_requests+0xfa/0x1a0
[   55.028003]  [<ffffffff813f26b3>] blk_mq_flush_plug_list+0x123/0x140
[   55.029710]  [<ffffffff813e7c77>] blk_flush_plug_list+0xa7/0x200
[   55.031363]  [<ffffffff813f1bec>] blk_sq_make_request+0x1dc/0x3b0
[   55.033025]  [<ffffffff813e5b0c>] ? generic_make_request_checks+0x24c/0x530
[   55.034829]  [<ffffffff813e6341>] generic_make_request+0xf1/0x1e0
[   55.036508]  [<ffffffff813e6497>] submit_bio+0x67/0x150
[   55.038034]  [<ffffffff8133cd61>] xfs_submit_ioend_bio+0x31/0x40
[   55.039797]  [<ffffffff8133ce51>] xfs_submit_ioend+0xe1/0x120
[   55.041410]  [<ffffffff8133e183>] xfs_vm_writepage+0x2a3/0x630
[   55.043038]  [<ffffffff811771a3>] __writepage+0x13/0x30
[   55.044568]  [<ffffffff811794bc>] write_cache_pages+0x22c/0x4c0
[   55.046214]  [<ffffffff81177190>] ? wb_position_ratio+0x1f0/0x1f0
[   55.047888]  [<ffffffff81179793>] generic_writepages+0x43/0x60
[   55.049520]  [<ffffffff8133d54e>] xfs_vm_writepages+0x4e/0x60
[   55.051130]  [<ffffffff8117a49e>] do_writepages+0x1e/0x30
[   55.052694]  [<ffffffff81216b05>] __writeback_single_inode+0x45/0x2e0
[   55.054410]  [<ffffffff812172e1>] writeback_sb_inodes+0x271/0x4d0
[   55.056075]  [<ffffffff812175c9>] __writeback_inodes_wb+0x89/0xc0
[   55.057749]  [<ffffffff812178ea>] wb_writeback+0x23a/0x2c0
[   55.059312]  [<ffffffff8121806f>] wb_workfn+0x20f/0x390
[   55.060846]  [<ffffffff81091257>] process_one_work+0x157/0x420
[   55.062466]  [<ffffffff81091eb9>] worker_thread+0x69/0x4a0
[   55.064030]  [<ffffffff81091e50>] ? rescuer_thread+0x380/0x380
[   55.065655]  [<ffffffff8109741f>] kthread+0xef/0x110
[   55.067141]  [<ffffffff81097330>] ? kthread_park+0x60/0x60
[   55.068713]  [<ffffffff818c5f8f>] ret_from_fork+0x3f/0x70
[   55.070255]  [<ffffffff81097330>] ? kthread_park+0x60/0x60
[   55.071830] Mem-Info:
[   55.072889] active_anon:5024 inactive_anon:1642 isolated_anon:0
[   55.072889]  active_file:22984 inactive_file:106547 isolated_file:25
[   55.072889]  unevictable:93684 dirty:7732 writeback:13026 unstable:0
[   55.072889]  slab_reclaimable:13082 slab_unreclaimable:3604
[   55.072889]  mapped:2866 shmem:1687 pagetables:876 bounce:0
[   55.072889]  free:1931 free_pcp:45 free_cma:0
[   55.081979] Node 0 DMA free:3948kB min:60kB low:72kB high:88kB active_anon:328kB inactive_anon:124kB active_file:1680kB inactive_file:1704kB unevictable:6936kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:52kB writeback:76kB mapped:440kB shmem:124kB slab_reclaimable:472kB slab_unreclaimable:232kB kernel_stack:80kB pagetables:84kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[   55.093512] lowmem_reserve[]: 0 972 972 972
[   55.095362] Node 0 DMA32 free:3776kB min:3824kB low:4780kB high:5736kB active_anon:19768kB inactive_anon:6444kB active_file:90256kB inactive_file:424484kB unevictable:367800kB isolated(anon):0kB isolated(file):100kB present:1032064kB managed:997492kB mlocked:0kB dirty:30876kB writeback:52028kB mapped:11024kB shmem:6624kB slab_reclaimable:51856kB slab_unreclaimable:14184kB kernel_stack:2864kB pagetables:3420kB unstable:0kB bounce:0kB free_pcp:180kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:100 all_unreclaimable? no
[   55.107443] lowmem_reserve[]: 0 0 0 0
[   55.109213] Node 0 DMA: 4*4kB (U) 3*8kB (UE) 2*16kB (UM) 1*32kB (E) 0*64kB 2*128kB (UM) 2*256kB (UM) 2*512kB (ME) 2*1024kB (ME) 0*2048kB 0*4096kB = 3944kB
[   55.114800] Node 0 DMA32: 90*4kB (UE) 13*8kB (UE) 10*16kB (UME) 7*32kB (UME) 46*64kB (ME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3792kB
[   55.120027] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[   55.122668] 224906 total pagecache pages
[   55.123990] 0 pages in swap cache
[   55.125207] Swap cache stats: add 0, delete 0, find 0/0
[   55.126738] Free swap  = 0kB
[   55.127890] Total swap = 0kB
[   55.129039] 262014 pages RAM
[   55.130190] 0 pages HighMem/MovableOnly
[   55.131495] 8664 pages reserved
[   55.132690] 0 pages hwpoisoned
[   55.133868] SLUB: Unable to allocate memory on node -1 (gfp=0x2080000)
[   55.135607]   cache: kmalloc-2048, object size: 2048, buffer size: 2048, default order: 3, min order: 0
[   55.138454]   node 0: slabs: 22, objs: 352, free: 0

Best Regards,
Huang, Ying

[toc] | [prev] | [next] | [standalone]

#1282902

FromMel Gorman <mgorman@techsingularity.net>
Date2015-12-03 11:20 +0100
Message-ID<qBz9v-3CM-11@gated-at.bofh.it>
In reply to#1282807
On Thu, Dec 03, 2015 at 04:46:53PM +0800, Huang, Ying wrote:
> Mel Gorman <mgorman@techsingularity.net> writes:
> 
> > On Wed, Dec 02, 2015 at 03:15:29PM +0100, Michal Hocko wrote:
> >> > > I didn't mention this allocation failure because I am not sure it is
> >> > > really related.
> >> > > 
> >> > 
> >> > I'm fairly sure it is. The failure is an allocation site that cannot
> >> > sleep but did not specify __GFP_HIGH.
> >> 
> >> yeah but this was the case even before your patch. As the caller used
> >> GFP_ATOMIC then it got __GFP_ATOMIC after your patch so it still
> >> managed to do ALLOC_HARDER. I would agree if this was an explicit
> >> GFP_NOWAIT. Unless I am missing something your patch hasn't changed the
> >> behavior for this particular allocation.
> >> 
> >
> > You're right. I think it's this hunk that is the problem.
> >
> > @@ -1186,7 +1186,7 @@ static struct request *blk_mq_map_request(struct
> > request_queue *q,
> >                 ctx = blk_mq_get_ctx(q);
> >                 hctx = q->mq_ops->map_queue(q, ctx->cpu);
> >                 blk_mq_set_alloc_data(&alloc_data, q,
> > -                               __GFP_WAIT|GFP_ATOMIC, false, ctx, hctx);
> > +                               __GFP_WAIT|__GFP_HIGH, false, ctx, hctx);
> >                 rq = __blk_mq_alloc_request(&alloc_data, rw);
> >                 ctx = alloc_data.ctx;
> >                 hctx = alloc_data.hctx;
> >
> > This specific path at this patch is not waking kswapd any more when it
> > should. A series of allocations there could hit the watermarks and never wake
> > kswapd and then be followed by an atomic allocation failure that woke kswapd.
> >
> > This bug gets fixed later by the commit 71baba4b92dc ("mm, page_alloc:
> > rename __GFP_WAIT to __GFP_RECLAIM") so it's not a bug in the current
> > kernel. However, it happens to break bisection and would be caught if each
> > individual commit was tested.
> >
> > Your __GFP_HIGH patch is still fine although not the direct fix for this
> > specific problem. Commit 71baba4b92dc is.
> >
> > Ying, does the page allocation failure messages happen when the whole
> > series is applied? i.e. is 4.4-rc3 ok?
> 
> There are allocation errors for 4.4-rc3 too. dmesg is attached.
> 

What is the result of the __GFP_HIGH patch to give it access to
reserves?

-- 
Mel Gorman
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1283538

From"Huang\, Ying" <ying.huang@linux.intel.com>
Date2015-12-04 03:00 +0100
Message-ID<qBNPb-4Dl-1@gated-at.bofh.it>
In reply to#1282902
Mel Gorman <mgorman@techsingularity.net> writes:

> On Thu, Dec 03, 2015 at 04:46:53PM +0800, Huang, Ying wrote:
>> Mel Gorman <mgorman@techsingularity.net> writes:
>> 
>> > On Wed, Dec 02, 2015 at 03:15:29PM +0100, Michal Hocko wrote:
>> >> > > I didn't mention this allocation failure because I am not sure it is
>> >> > > really related.
>> >> > > 
>> >> > 
>> >> > I'm fairly sure it is. The failure is an allocation site that cannot
>> >> > sleep but did not specify __GFP_HIGH.
>> >> 
>> >> yeah but this was the case even before your patch. As the caller used
>> >> GFP_ATOMIC then it got __GFP_ATOMIC after your patch so it still
>> >> managed to do ALLOC_HARDER. I would agree if this was an explicit
>> >> GFP_NOWAIT. Unless I am missing something your patch hasn't changed the
>> >> behavior for this particular allocation.
>> >> 
>> >
>> > You're right. I think it's this hunk that is the problem.
>> >
>> > @@ -1186,7 +1186,7 @@ static struct request *blk_mq_map_request(struct
>> > request_queue *q,
>> >                 ctx = blk_mq_get_ctx(q);
>> >                 hctx = q->mq_ops->map_queue(q, ctx->cpu);
>> >                 blk_mq_set_alloc_data(&alloc_data, q,
>> > -                               __GFP_WAIT|GFP_ATOMIC, false, ctx, hctx);
>> > +                               __GFP_WAIT|__GFP_HIGH, false, ctx, hctx);
>> >                 rq = __blk_mq_alloc_request(&alloc_data, rw);
>> >                 ctx = alloc_data.ctx;
>> >                 hctx = alloc_data.hctx;
>> >
>> > This specific path at this patch is not waking kswapd any more when it
>> > should. A series of allocations there could hit the watermarks and never wake
>> > kswapd and then be followed by an atomic allocation failure that woke kswapd.
>> >
>> > This bug gets fixed later by the commit 71baba4b92dc ("mm, page_alloc:
>> > rename __GFP_WAIT to __GFP_RECLAIM") so it's not a bug in the current
>> > kernel. However, it happens to break bisection and would be caught if each
>> > individual commit was tested.
>> >
>> > Your __GFP_HIGH patch is still fine although not the direct fix for this
>> > specific problem. Commit 71baba4b92dc is.
>> >
>> > Ying, does the page allocation failure messages happen when the whole
>> > series is applied? i.e. is 4.4-rc3 ok?
>> 
>> There are allocation errors for 4.4-rc3 too. dmesg is attached.
>> 
>
> What is the result of the __GFP_HIGH patch to give it access to
> reserves?

Applied Michal's patch on v4.4-rc3 and tested again, now there is no
page allocation failure.

Best Regards,
Huang, Ying
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [next] | [standalone]

#1285759

FromMichal Hocko <mhocko@kernel.org>
Date2015-12-07 17:20 +0100
Message-ID<qD6G8-6S6-61@gated-at.bofh.it>
In reply to#1283538
On Fri 04-12-15 09:53:35, Huang, Ying wrote:
> Mel Gorman <mgorman@techsingularity.net> writes:
[...]
> > What is the result of the __GFP_HIGH patch to give it access to
> > reserves?
> 
> Applied Michal's patch on v4.4-rc3 and tested again, now there is no
> page allocation failure.

I still think this is just a coincidence and __GFP_HIGH just
papers over it (it still makes sense for other reasons though). It
would be better imho to log all allocations which do not include
___GFP_KSWAPD_RECLAIM. This might help us to pin point a caller which
was previously waking kswapd up but it doesn't do so anymore.  There
would have to be a very specific pattern to trigger this because most
allocations would simply wake kswapd up. I am just wondering that
any debugging would simply alter the timing and/or allocation pattern
and we would see a different results so I am not sure this is worth
spending a lot of time on.

-- 
Michal Hocko
SUSE Labs
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web