Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1492231 > unrolled thread

Regression in mobility grouping?

Back to article view | Back to linux.kernel

Contents

  Regression in mobility grouping? Johannes Weiner <hannes@cmpxchg.org> - 2016-09-28 03:50 +0200
    Re: Regression in mobility grouping? Vlastimil Babka <vbabka@suse.cz> - 2016-09-28 11:10 +0200
      Re: Regression in mobility grouping? Johannes Weiner <hannes@cmpxchg.org> - 2016-09-28 17:50 +0200
        Re: Regression in mobility grouping? Johannes Weiner <hannes@cmpxchg.org> - 2016-09-29 04:30 +0200
          Re: Regression in mobility grouping? Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2016-09-29 08:10 +0200
            Re: Regression in mobility grouping? Johannes Weiner <hannes@cmpxchg.org> - 2016-09-29 18:20 +0200
              Re: Regression in mobility grouping? Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2016-10-13 09:40 +0200
          Re: Regression in mobility grouping? Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 09:20 +0200
    Re: Regression in mobility grouping? Mel Gorman <mgorman@suse.de> - 2016-09-28 12:30 +0200
      Re: Regression in mobility grouping? Johannes Weiner <hannes@cmpxchg.org> - 2016-09-28 18:50 +0200
    [RFC 2/4] mm, compaction: add migratetype to compact_control Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 23:10 +0200
    [RFC 1/4] mm, compaction: change migrate_async_suitable() to suitable_migration_source() Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 23:10 +0200
    [RFC 3/4] mm, compaction: restrict async compaction to matching migratetype Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 23:10 +0200
    [RFC 4/4] mm, page_alloc: disallow migratetype fallback in fastpath Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 23:10 +0200
      Re: [RFC 4/4] mm, page_alloc: disallow migratetype fallback in  fastpath Vlastimil Babka <vbabka@suse.cz> - 2016-10-12 17:20 +0200
      Re: [RFC 4/4] mm, page_alloc: disallow migratetype fallback in  fastpath Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2016-10-13 10:00 +0200
        Re: [RFC 4/4] mm, page_alloc: disallow migratetype fallback in  fastpath Vlastimil Babka <vbabka@suse.cz> - 2016-10-13 13:50 +0200
    [RFC 0/4] try to reduce fragmenting fallbacks Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 23:10 +0200
      [RFC 5/4] mm, page_alloc: split smallest stolen page in fallback Vlastimil Babka <vbabka@suse.cz> - 2016-10-07 10:40 +0200
      Re: [RFC 0/4] try to reduce fragmenting fallbacks Johannes Weiner <hannes@cmpxchg.org> - 2016-10-10 19:30 +0200
      [RFC 6/4] mm, page_alloc: introduce MIGRATE_MIXED migratetype Vlastimil Babka <vbabka@suse.cz> - 2016-10-11 15:20 +0200
      [RFC 7/4] mm, page_alloc: count movable pages when stealing Vlastimil Babka <vbabka@suse.cz> - 2016-10-13 16:20 +0200

Page 1 of 2  [1] 2  Next page →

#1492231 — Regression in mobility grouping?

[Multipart message — attachments visible in raw view] — view raw

Hi guys,

we noticed what looks like a regression in page mobility grouping
during an upgrade from 3.10 to 4.0. Identical machines, workloads, and
uptime, but /proc/pagetypeinfo on 3.10 looks like this:

Number of blocks type     Unmovable  Reclaimable      Movable      Reserve      Isolate 
Node 1, zone   Normal          815          433        31518            2            0 

and on 4.0 like this:

Number of blocks type     Unmovable  Reclaimable      Movable      Reserve          CMA      Isolate 
Node 1, zone   Normal         3880         3530        25356            2            0            0 

4.0 is either polluting pageblocks more aggressively at allocation, or
is not able to make pageblocks movable again when the reclaimable and
unmovable allocations are released. Invoking compaction manually
(/proc/sys/vm/compact_memory) is not bringing them back, either.

The problem we are debugging is that these machines have a very high
rate of order-3 allocations (fdtable during fork, network rx), and
after the upgrade allocstalls have increased dramatically. I'm not
entirely sure this is the same issue, since even order-0 allocations
are struggling, but the mobility grouping in itself looks problematic.

I'm still going through the changes relevant to mobility grouping in
that timeframe, but if this rings a bell for anyone, it would help. I
hate blaming random patches, but these caught my eye:

9c0415e mm: more aggressive page stealing for UNMOVABLE allocations
3a1086f mm: always steal split buddies in fallback allocations
99592d5 mm: when stealing freepages, also take pages created by splitting buddy page

The changelog states that by aggressively stealing split buddy pages
during a fallback allocation we avoid subsequent stealing. But since
there are generally more movable/reclaimable pages available, and so
less falling back and stealing freepages on behalf of movable, won't
this mean that we could expect exactly that result - growing numbers
of unmovable blocks, while rarely stealing them back in movable alloc
fallbacks? And the expansion of !MOVABLE blocks would over time make
compaction less and less effective too, seeing as it doesn't consider
anything !MOVABLE suitable migration targets?

Attached are the full /proc/pagetypeinfo and /proc/buddyinfo from both
kernels on machines with similar uptimes and directly after invoking
compaction. As you can see, the buddy lists are much more fragmented
on 4.0, with unmovable/reclaimable allocations polluting more blocks.

Any thoughts on this would be greatly appreciated. I can test patches.

Thanks!

[toc] | [next] | [standalone]

#1492370

On 09/28/2016 03:41 AM, Johannes Weiner wrote:
> Hi guys,
> 
> we noticed what looks like a regression in page mobility grouping
> during an upgrade from 3.10 to 4.0. Identical machines, workloads, and
> uptime, but /proc/pagetypeinfo on 3.10 looks like this:
> 
> Number of blocks type     Unmovable  Reclaimable      Movable      Reserve      Isolate 
> Node 1, zone   Normal          815          433        31518            2            0 
> 
> and on 4.0 like this:
> 
> Number of blocks type     Unmovable  Reclaimable      Movable      Reserve          CMA      Isolate 
> Node 1, zone   Normal         3880         3530        25356            2            0            0 

It's worth to keep in mind that this doesn't reflect where the actual
unmovable pages reside. It might be that in 3.10 they are spread within
the movable pages. IIRC enabling page_owner (not sure if in 4.0, there
were some later fixes I think) can augment pagetypeinfo with at least
some statistics of polluted pageblocks.

Does e.g. /proc/meminfo suggest how much unmovable/reclaimable memory
there should be allocated and if it would fill the respective
pageblocks, or if they are poorly utilized?

> 4.0 is either polluting pageblocks more aggressively at allocation, or
> is not able to make pageblocks movable again when the reclaimable and
> unmovable allocations are released. Invoking compaction manually
> (/proc/sys/vm/compact_memory) is not bringing them back, either.
>
> The problem we are debugging is that these machines have a very high
> rate of order-3 allocations (fdtable during fork, network rx), and
> after the upgrade allocstalls have increased dramatically. I'm not
> entirely sure this is the same issue, since even order-0 allocations
> are struggling, but the mobility grouping in itself looks problematic.
> 
> I'm still going through the changes relevant to mobility grouping in
> that timeframe, but if this rings a bell for anyone, it would help. I
> hate blaming random patches, but these caught my eye:
> 
> 9c0415e mm: more aggressive page stealing for UNMOVABLE allocations
> 3a1086f mm: always steal split buddies in fallback allocations
> 99592d5 mm: when stealing freepages, also take pages created by splitting buddy page

Check also the changelogs for mentions of earlier commits, e.g. 99592d5
should be restoring behavior that changed in 3.12-3.13 and you are
upgrading from 3.10.

> The changelog states that by aggressively stealing split buddy pages
> during a fallback allocation we avoid subsequent stealing. But since
> there are generally more movable/reclaimable pages available, and so
> less falling back and stealing freepages on behalf of movable, won't
> this mean that we could expect exactly that result - growing numbers
> of unmovable blocks, while rarely stealing them back in movable alloc
> fallbacks? And the expansion of !MOVABLE blocks would over time make
> compaction less and less effective too, seeing as it doesn't consider
> anything !MOVABLE suitable migration targets?

Yeah this is an issue with compaction that was brought up recently and I
want to tackle next.

> Attached are the full /proc/pagetypeinfo and /proc/buddyinfo from both
> kernels on machines with similar uptimes and directly after invoking
> compaction. As you can see, the buddy lists are much more fragmented
> on 4.0, with unmovable/reclaimable allocations polluting more blocks.
> 
> Any thoughts on this would be greatly appreciated. I can test patches.

I guess testing revert of 9c0415e could give us some idea. Commit
3a1086f shouldn't result in pageblock marking differences and as I said
above, 99592d5 should be just restoring to what 3.10 did.

[toc] | [prev] | [next] | [standalone]

#1492666

Hi Vlastimil,

On Wed, Sep 28, 2016 at 11:00:15AM +0200, Vlastimil Babka wrote:
> On 09/28/2016 03:41 AM, Johannes Weiner wrote:
> > Hi guys,
> > 
> > we noticed what looks like a regression in page mobility grouping
> > during an upgrade from 3.10 to 4.0. Identical machines, workloads, and
> > uptime, but /proc/pagetypeinfo on 3.10 looks like this:
> > 
> > Number of blocks type     Unmovable  Reclaimable      Movable      Reserve      Isolate 
> > Node 1, zone   Normal          815          433        31518            2            0 
> > 
> > and on 4.0 like this:
> > 
> > Number of blocks type     Unmovable  Reclaimable      Movable      Reserve          CMA      Isolate 
> > Node 1, zone   Normal         3880         3530        25356            2            0            0 
> 
> It's worth to keep in mind that this doesn't reflect where the actual
> unmovable pages reside. It might be that in 3.10 they are spread within
> the movable pages. IIRC enabling page_owner (not sure if in 4.0, there
> were some later fixes I think) can augment pagetypeinfo with at least
> some statistics of polluted pageblocks.

Thanks, I'll look at the mixed block counts. I failed to make clear,
we saw that issue in the switch from 3.10 to 4.0, and I mentioned
those two kernels as last known good / first known bad. But later
kernels - we tried with 4.6 - look the same. This appears to be a
regression in (higher-order) allocation service quality somewhere
after 3.10 that persists into current kernels.

> Does e.g. /proc/meminfo suggest how much unmovable/reclaimable memory
> there should be allocated and if it would fill the respective
> pageblocks, or if they are poorly utilized?

They are very poorly utilized. On a machine with 90% anon/cache pages
alone we saw 50% of the page blocks unmovable.

> > 4.0 is either polluting pageblocks more aggressively at allocation, or
> > is not able to make pageblocks movable again when the reclaimable and
> > unmovable allocations are released. Invoking compaction manually
> > (/proc/sys/vm/compact_memory) is not bringing them back, either.
> >
> > The problem we are debugging is that these machines have a very high
> > rate of order-3 allocations (fdtable during fork, network rx), and
> > after the upgrade allocstalls have increased dramatically. I'm not
> > entirely sure this is the same issue, since even order-0 allocations
> > are struggling, but the mobility grouping in itself looks problematic.
> > 
> > I'm still going through the changes relevant to mobility grouping in
> > that timeframe, but if this rings a bell for anyone, it would help. I
> > hate blaming random patches, but these caught my eye:
> > 
> > 9c0415e mm: more aggressive page stealing for UNMOVABLE allocations
> > 3a1086f mm: always steal split buddies in fallback allocations
> > 99592d5 mm: when stealing freepages, also take pages created by splitting buddy page
> 
> Check also the changelogs for mentions of earlier commits, e.g. 99592d5
> should be restoring behavior that changed in 3.12-3.13 and you are
> upgrading from 3.10.

Good point.

> > The changelog states that by aggressively stealing split buddy pages
> > during a fallback allocation we avoid subsequent stealing. But since
> > there are generally more movable/reclaimable pages available, and so
> > less falling back and stealing freepages on behalf of movable, won't
> > this mean that we could expect exactly that result - growing numbers
> > of unmovable blocks, while rarely stealing them back in movable alloc
> > fallbacks? And the expansion of !MOVABLE blocks would over time make
> > compaction less and less effective too, seeing as it doesn't consider
> > anything !MOVABLE suitable migration targets?
> 
> Yeah this is an issue with compaction that was brought up recently and I
> want to tackle next.

Agreed, it would be nice if compaction could reclaim unmovable and
reclaimable blocks whose polluting allocations have since been freed.

But there is a limit to how lazy mobility grouping can be and still
expect compaction to fix it up. If 50% of the page blocks are marked
unmovable, we don't pack incoming polluting allocations. When spread
out the right way, even just a few of those can have a devastating
impact on overall compactability.

So regardless of future compaction improvements, we need to get
anti-frag accuracy in the allocator closer to 3.10 levels again.

> > Attached are the full /proc/pagetypeinfo and /proc/buddyinfo from both
> > kernels on machines with similar uptimes and directly after invoking
> > compaction. As you can see, the buddy lists are much more fragmented
> > on 4.0, with unmovable/reclaimable allocations polluting more blocks.
> > 
> > Any thoughts on this would be greatly appreciated. I can test patches.
> 
> I guess testing revert of 9c0415e could give us some idea. Commit
> 3a1086f shouldn't result in pageblock marking differences and as I said
> above, 99592d5 should be just restoring to what 3.10 did.

I can give this a shot, but note that this commit makes only unmovable
stealing more aggressive. We see reclaimable blocks up as well.

The workload is fairly variable, so it'll take about a day to smooth
out a meaningful average.

Thanks for your insights, Vlastimil!

[toc] | [prev] | [next] | [standalone]

#1493110

On Wed, Sep 28, 2016 at 11:39:25AM -0400, Johannes Weiner wrote:
> On Wed, Sep 28, 2016 at 11:00:15AM +0200, Vlastimil Babka wrote:
> > I guess testing revert of 9c0415e could give us some idea. Commit
> > 3a1086f shouldn't result in pageblock marking differences and as I said
> > above, 99592d5 should be just restoring to what 3.10 did.
> 
> I can give this a shot, but note that this commit makes only unmovable
> stealing more aggressive. We see reclaimable blocks up as well.

Quick update, I reverted back to stealing eagerly only on behalf of
MIGRATE_RECLAIMABLE allocations in a 4.6 kernel:

static bool can_steal_fallback(unsigned int order, int start_mt)
{
        if (order >= pageblock_order / 2 ||
            start_mt == MIGRATE_RECLAIMABLE ||
            page_group_by_mobility_disabled)
                return true;

        return false;
}

Yet, I still see UNMOVABLE growing to the thousands within minutes,
whereas 3.10 didn't reach those numbers even after days of uptime.

Okay, that wasn't it. However, there is something fishy going on,
because I see extfrag traces like these:

<idle>-0     [006] d.s.  1110.217281: mm_page_alloc_extfrag: page=ffffea0064142000 pfn=26235008 alloc_order=3 fallback_order=3 pageblock_order=9 alloc_migratetype=0 fallback_migratetype=2 fragmenting=1 change_ownership=1

enum {
        MIGRATE_UNMOVABLE,
        MIGRATE_MOVABLE,
        MIGRATE_RECLAIMABLE,
        MIGRATE_PCPTYPES,       /* the number of types on the pcp lists */
        MIGRATE_HIGHATOMIC = MIGRATE_PCPTYPES,
	...
};

This is an UNMOVABLE order-3 allocation falling back to RECLAIMABLE.
According to can_steal_fallback(), this allocation shouldn't steal the
pageblock, yet change_ownership=1 indicates the block is UNMOVABLE.

Who converted it? I wonder if there is a bug in ownership management,
and there was an UNMOVABLE block on the RECLAIMABLE freelist from the
beginning. AFAICS we never validate list/mt consistency anywhere.

I'll continue looking tomorrow.

[toc] | [prev] | [next] | [standalone]

#1493147

On Wed, Sep 28, 2016 at 10:25:40PM -0400, Johannes Weiner wrote:
> On Wed, Sep 28, 2016 at 11:39:25AM -0400, Johannes Weiner wrote:
> > On Wed, Sep 28, 2016 at 11:00:15AM +0200, Vlastimil Babka wrote:
> > > I guess testing revert of 9c0415e could give us some idea. Commit
> > > 3a1086f shouldn't result in pageblock marking differences and as I said
> > > above, 99592d5 should be just restoring to what 3.10 did.
> > 
> > I can give this a shot, but note that this commit makes only unmovable
> > stealing more aggressive. We see reclaimable blocks up as well.
> 
> Quick update, I reverted back to stealing eagerly only on behalf of
> MIGRATE_RECLAIMABLE allocations in a 4.6 kernel:

Hello, Johannes.

I think that it would be better to check 3.10 with above patches.
Fragmentation depends on not only policy itself but also
allocation/free pattern. There might be a large probability that
allocation/free pattern is changed in this large kernel version
difference.

> 
> static bool can_steal_fallback(unsigned int order, int start_mt)
> {
>         if (order >= pageblock_order / 2 ||
>             start_mt == MIGRATE_RECLAIMABLE ||
>             page_group_by_mobility_disabled)
>                 return true;
> 
>         return false;
> }
> 
> Yet, I still see UNMOVABLE growing to the thousands within minutes,
> whereas 3.10 didn't reach those numbers even after days of uptime.
> 
> Okay, that wasn't it. However, there is something fishy going on,
> because I see extfrag traces like these:
> 
> <idle>-0     [006] d.s.  1110.217281: mm_page_alloc_extfrag: page=ffffea0064142000 pfn=26235008 alloc_order=3 fallback_order=3 pageblock_order=9 alloc_migratetype=0 fallback_migratetype=2 fragmenting=1 change_ownership=1
> 
> enum {
>         MIGRATE_UNMOVABLE,
>         MIGRATE_MOVABLE,
>         MIGRATE_RECLAIMABLE,
>         MIGRATE_PCPTYPES,       /* the number of types on the pcp lists */
>         MIGRATE_HIGHATOMIC = MIGRATE_PCPTYPES,
> 	...
> };
> 
> This is an UNMOVABLE order-3 allocation falling back to RECLAIMABLE.
> According to can_steal_fallback(), this allocation shouldn't steal the
> pageblock, yet change_ownership=1 indicates the block is UNMOVABLE.
> 
> Who converted it? I wonder if there is a bug in ownership management,
> and there was an UNMOVABLE block on the RECLAIMABLE freelist from the
> beginning. AFAICS we never validate list/mt consistency anywhere.

According to my code review, it would be possible. When stealing
happens, we moved those buddy pages to current requested migratetype
buddy list. If the other migratetype allocation request comes and
stealing from the buddy list of previous requested migratetype
happens, change_ownership will show '1' even if there is no ownership
changing.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1493555

On Thu, Sep 29, 2016 at 03:14:33PM +0900, Joonsoo Kim wrote:
> On Wed, Sep 28, 2016 at 10:25:40PM -0400, Johannes Weiner wrote:
> > On Wed, Sep 28, 2016 at 11:39:25AM -0400, Johannes Weiner wrote:
> > > On Wed, Sep 28, 2016 at 11:00:15AM +0200, Vlastimil Babka wrote:
> > > > I guess testing revert of 9c0415e could give us some idea. Commit
> > > > 3a1086f shouldn't result in pageblock marking differences and as I said
> > > > above, 99592d5 should be just restoring to what 3.10 did.
> > > 
> > > I can give this a shot, but note that this commit makes only unmovable
> > > stealing more aggressive. We see reclaimable blocks up as well.
> > 
> > Quick update, I reverted back to stealing eagerly only on behalf of
> > MIGRATE_RECLAIMABLE allocations in a 4.6 kernel:
> 
> Hello, Johannes.
> 
> I think that it would be better to check 3.10 with above patches.
> Fragmentation depends on not only policy itself but also
> allocation/free pattern. There might be a large probability that
> allocation/free pattern is changed in this large kernel version
> difference.

You mean backport suspicious patches to 3.10 until I can reproduce it
there? I'm not sure. You're correct, the patterns very likely *have*
changed. But that alone cannot explain mobility grouping breaking that
badly. There is a reproducable bad behavior. It should be easier to
track down than to try to recreate it in the last-known-good kernel.

> > This is an UNMOVABLE order-3 allocation falling back to RECLAIMABLE.
> > According to can_steal_fallback(), this allocation shouldn't steal the
> > pageblock, yet change_ownership=1 indicates the block is UNMOVABLE.
> > 
> > Who converted it? I wonder if there is a bug in ownership management,
> > and there was an UNMOVABLE block on the RECLAIMABLE freelist from the
> > beginning. AFAICS we never validate list/mt consistency anywhere.
> 
> According to my code review, it would be possible. When stealing
> happens, we moved those buddy pages to current requested migratetype
> buddy list. If the other migratetype allocation request comes and
> stealing from the buddy list of previous requested migratetype
> happens, change_ownership will show '1' even if there is no ownership
> changing.

These two paths should exclude each other through the zone->lock, no?

[toc] | [prev] | [next] | [standalone]

#1500068

Sorry for late response.

On Thu, Sep 29, 2016 at 12:14:02PM -0400, Johannes Weiner wrote:
> On Thu, Sep 29, 2016 at 03:14:33PM +0900, Joonsoo Kim wrote:
> > On Wed, Sep 28, 2016 at 10:25:40PM -0400, Johannes Weiner wrote:
> > > On Wed, Sep 28, 2016 at 11:39:25AM -0400, Johannes Weiner wrote:
> > > > On Wed, Sep 28, 2016 at 11:00:15AM +0200, Vlastimil Babka wrote:
> > > > > I guess testing revert of 9c0415e could give us some idea. Commit
> > > > > 3a1086f shouldn't result in pageblock marking differences and as I said
> > > > > above, 99592d5 should be just restoring to what 3.10 did.
> > > > 
> > > > I can give this a shot, but note that this commit makes only unmovable
> > > > stealing more aggressive. We see reclaimable blocks up as well.
> > > 
> > > Quick update, I reverted back to stealing eagerly only on behalf of
> > > MIGRATE_RECLAIMABLE allocations in a 4.6 kernel:
> > 
> > Hello, Johannes.
> > 
> > I think that it would be better to check 3.10 with above patches.
> > Fragmentation depends on not only policy itself but also
> > allocation/free pattern. There might be a large probability that
> > allocation/free pattern is changed in this large kernel version
> > difference.
> 
> You mean backport suspicious patches to 3.10 until I can reproduce it
> there? I'm not sure. You're correct, the patterns very likely *have*
> changed. But that alone cannot explain mobility grouping breaking that
> badly. There is a reproducable bad behavior. It should be easier to
> track down than to try to recreate it in the last-known-good kernel.

Okay. It is just my two cents.

> 
> > > This is an UNMOVABLE order-3 allocation falling back to RECLAIMABLE.
> > > According to can_steal_fallback(), this allocation shouldn't steal the
> > > pageblock, yet change_ownership=1 indicates the block is UNMOVABLE.
> > > 
> > > Who converted it? I wonder if there is a bug in ownership management,
> > > and there was an UNMOVABLE block on the RECLAIMABLE freelist from the
> > > beginning. AFAICS we never validate list/mt consistency anywhere.
> > 
> > According to my code review, it would be possible. When stealing
> > happens, we moved those buddy pages to current requested migratetype
> > buddy list. If the other migratetype allocation request comes and
> > stealing from the buddy list of previous requested migratetype
> > happens, change_ownership will show '1' even if there is no ownership
> > changing.
> 
> These two paths should exclude each other through the zone->lock, no?

zone->lock ensures that changing migratetype of pageblock happens
sequentially. But, it doesn't protect where actual freepage of some
pageblock is attached. For example, freepage on unmovable pageblock
could be attached on the movable buddy list and wrong information
about change_ownership=1 would be possible.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1493175

On 09/29/2016 04:25 AM, Johannes Weiner wrote:
> On Wed, Sep 28, 2016 at 11:39:25AM -0400, Johannes Weiner wrote:
>> On Wed, Sep 28, 2016 at 11:00:15AM +0200, Vlastimil Babka wrote:
>>> I guess testing revert of 9c0415e could give us some idea. Commit
>>> 3a1086f shouldn't result in pageblock marking differences and as I said
>>> above, 99592d5 should be just restoring to what 3.10 did.
>>
>> I can give this a shot, but note that this commit makes only unmovable
>> stealing more aggressive. We see reclaimable blocks up as well.
>
> Quick update, I reverted back to stealing eagerly only on behalf of
> MIGRATE_RECLAIMABLE allocations in a 4.6 kernel:
>
> static bool can_steal_fallback(unsigned int order, int start_mt)
> {
>         if (order >= pageblock_order / 2 ||
>             start_mt == MIGRATE_RECLAIMABLE ||
>             page_group_by_mobility_disabled)
>                 return true;
>
>         return false;
> }
>
> Yet, I still see UNMOVABLE growing to the thousands within minutes,
> whereas 3.10 didn't reach those numbers even after days of uptime.
>
> Okay, that wasn't it. However, there is something fishy going on,
> because I see extfrag traces like these:
>
> <idle>-0     [006] d.s.  1110.217281: mm_page_alloc_extfrag: page=ffffea0064142000 pfn=26235008 alloc_order=3 fallback_order=3 pageblock_order=9 alloc_migratetype=0 fallback_migratetype=2 fragmenting=1 change_ownership=1
>
> enum {
>         MIGRATE_UNMOVABLE,
>         MIGRATE_MOVABLE,
>         MIGRATE_RECLAIMABLE,
>         MIGRATE_PCPTYPES,       /* the number of types on the pcp lists */
>         MIGRATE_HIGHATOMIC = MIGRATE_PCPTYPES,
> 	...
> };
>
> This is an UNMOVABLE order-3 allocation falling back to RECLAIMABLE.
> According to can_steal_fallback(), this allocation shouldn't steal the
> pageblock, yet change_ownership=1 indicates the block is UNMOVABLE.
>
> Who converted it? I wonder if there is a bug in ownership management,
> and there was an UNMOVABLE block on the RECLAIMABLE freelist from the
> beginning. AFAICS we never validate list/mt consistency anywhere.

Hm yes there are e.g. no strong guarantees for pageblock migratetype and 
relevant pages being on freelist of the same type, except for ISOLATE, 
for performance reasons. IIRC pageblock type is checked when putting a 
page on pcplist and then it may diverge before it's flushed on freelist. 
So it's possible the fallback page was on RECLAIMABLE list
while the pageblock was marked as UNMOVABLE.

Also the tracepoint is racy so that steal_suitable_fallback() doesn't 
have to communicate back whether it was truly stealing whole pageblock.

> I'll continue looking tomorrow.
>

[toc] | [prev] | [next] | [standalone]

#1492517

On Tue, Sep 27, 2016 at 09:41:48PM -0400, Johannes Weiner wrote:
> Hi guys,
> 
> we noticed what looks like a regression in page mobility grouping
> during an upgrade from 3.10 to 4.0. Identical machines, workloads, and
> uptime, but /proc/pagetypeinfo on 3.10 looks like this:
> 
> Number of blocks type     Unmovable  Reclaimable      Movable      Reserve      Isolate 
> Node 1, zone   Normal          815          433        31518            2            0 
> 
> and on 4.0 like this:
> 
> Number of blocks type     Unmovable  Reclaimable      Movable      Reserve          CMA      Isolate 
> Node 1, zone   Normal         3880         3530        25356            2            0            0 
> 

Unmovable pageblocks is not necessarily related to the number of
unmovable pages in the system although it is obviously a concern.
Basically there are two usual approaches to investigating this -- close
attention to the extfrag tracepoint and analysing high-order allocation
failures.

It's drastic, but when migration grouping was first implemented it was
necessary to use a variation of PAGE_OWNER to walk the movable pageblocks
identifying unmovable allocations in there. I also used to have a
debugging patch that would print out the owner of all pages that failed
to migrate within an unmovable block. Unfortunately I don't have these
patches any more and they wouldn't apply anyway but it'd be easier to
implement today than it was 7-8 years ago.

> 4.0 is either polluting pageblocks more aggressively at allocation, or
> is not able to make pageblocks movable again when the reclaimable and
> unmovable allocations are released. Invoking compaction manually
> (/proc/sys/vm/compact_memory) is not bringing them back, either.
> 
> The problem we are debugging is that these machines have a very high
> rate of order-3 allocations (fdtable during fork, network rx), and
> after the upgrade allocstalls have increased dramatically. I'm not
> entirely sure this is the same issue, since even order-0 allocations
> are struggling, but the mobility grouping in itself looks problematic.
> 

Network RX is likely to be atomic allocations. Another potentially place
to focus on is the use of HighAtomic pageblocks and either increasing
them in size or protecting them more aggressively.

> I'm still going through the changes relevant to mobility grouping in
> that timeframe, but if this rings a bell for anyone, it would help. I
> hate blaming random patches, but these caught my eye:
> 
> 9c0415e mm: more aggressive page stealing for UNMOVABLE allocations
> 3a1086f mm: always steal split buddies in fallback allocations
> 99592d5 mm: when stealing freepages, also take pages created by splitting buddy page
> 
> The changelog states that by aggressively stealing split buddy pages
> during a fallback allocation we avoid subsequent stealing. But since
> there are generally more movable/reclaimable pages available, and so
> less falling back and stealing freepages on behalf of movable, won't
> this mean that we could expect exactly that result - growing numbers
> of unmovable blocks, while rarely stealing them back in movable alloc
> fallbacks? And the expansion of !MOVABLE blocks would over time make
> compaction less and less effective too, seeing as it doesn't consider
> anything !MOVABLE suitable migration targets?
> 

It's a solid theory. There has been a lot of activity to weaken fragmentation
avoidance protection to reduce latency. Unfortunately external fragmentation
continues to be one of those topics that is very difficult to precisely
define because it's a matter of definition whether it's important or
not.

Another avenue worth considering is that compaction used to scan unmovable
pageblocks and migrate movable pages out of there but that was weakened
over time trying to allocate THP pages from direct allocation context
quickly enough. I'm not exactly sure what we do there at the moment and
whether kcompactd cleans unmovable pageblocks or not. It takes time but
it also reduces unmovable pageblock steals over time (or at least it did
a few years ago when I last investigated this in depth).

Unfortunately I do not have any suggestions offhand on how it could be
easily improved without going back to first principals and identifying
what pages end up in awkward positions, why and whether the cost of
"cleaning" unmovable pageblocks during compaction for a high-order
allocation is justified or not.

-- 
Mel Gorman
SUSE Labs

[toc] | [prev] | [next] | [standalone]

#1492760

Hi Mel,

On Wed, Sep 28, 2016 at 11:26:09AM +0100, Mel Gorman wrote:
> On Tue, Sep 27, 2016 at 09:41:48PM -0400, Johannes Weiner wrote:
> > Hi guys,
> > 
> > we noticed what looks like a regression in page mobility grouping
> > during an upgrade from 3.10 to 4.0. Identical machines, workloads, and
> > uptime, but /proc/pagetypeinfo on 3.10 looks like this:
> > 
> > Number of blocks type     Unmovable  Reclaimable      Movable      Reserve      Isolate 
> > Node 1, zone   Normal          815          433        31518            2            0 
> > 
> > and on 4.0 like this:
> > 
> > Number of blocks type     Unmovable  Reclaimable      Movable      Reserve          CMA      Isolate 
> > Node 1, zone   Normal         3880         3530        25356            2            0            0 
> > 
> 
> Unmovable pageblocks is not necessarily related to the number of
> unmovable pages in the system although it is obviously a concern.
> Basically there are two usual approaches to investigating this -- close
> attention to the extfrag tracepoint and analysing high-order allocation
> failures.
> 
> It's drastic, but when migration grouping was first implemented it was
> necessary to use a variation of PAGE_OWNER to walk the movable pageblocks
> identifying unmovable allocations in there. I also used to have a
> debugging patch that would print out the owner of all pages that failed
> to migrate within an unmovable block. Unfortunately I don't have these
> patches any more and they wouldn't apply anyway but it'd be easier to
> implement today than it was 7-8 years ago.

I've stared at the extfrag tracepoint for a while, and there really is
a high rate of block conversion going on, even after some uptime. But
it's not entirely obvious why. You'd think with large parts of memory
already in unmovable blocks - and we know them to be sparse based on
consumer breakdown in /proc/meminfo - there should be enough existing
blocks to choose from.

The PAGE_OWNER part of /proc/pagetypeinfo should be a good start for
seeing how efficiently we're packing by type. Thanks, I'll check that.

> > 4.0 is either polluting pageblocks more aggressively at allocation, or
> > is not able to make pageblocks movable again when the reclaimable and
> > unmovable allocations are released. Invoking compaction manually
> > (/proc/sys/vm/compact_memory) is not bringing them back, either.
> > 
> > The problem we are debugging is that these machines have a very high
> > rate of order-3 allocations (fdtable during fork, network rx), and
> > after the upgrade allocstalls have increased dramatically. I'm not
> > entirely sure this is the same issue, since even order-0 allocations
> > are struggling, but the mobility grouping in itself looks problematic.
> > 
> 
> Network RX is likely to be atomic allocations. Another potentially place
> to focus on is the use of HighAtomic pageblocks and either increasing
> them in size or protecting them more aggressively.

That's a good point in general for these machines and their workloads,
since we push them pretty hard with a combination of high memory
utilization and heavy network traffic with large packets sizes.

But note that MIGRATE_HIGHATOMIC was introduced only after the first
bad kernel.

> > I'm still going through the changes relevant to mobility grouping in
> > that timeframe, but if this rings a bell for anyone, it would help. I
> > hate blaming random patches, but these caught my eye:
> > 
> > 9c0415e mm: more aggressive page stealing for UNMOVABLE allocations
> > 3a1086f mm: always steal split buddies in fallback allocations
> > 99592d5 mm: when stealing freepages, also take pages created by splitting buddy page
> > 
> > The changelog states that by aggressively stealing split buddy pages
> > during a fallback allocation we avoid subsequent stealing. But since
> > there are generally more movable/reclaimable pages available, and so
> > less falling back and stealing freepages on behalf of movable, won't
> > this mean that we could expect exactly that result - growing numbers
> > of unmovable blocks, while rarely stealing them back in movable alloc
> > fallbacks? And the expansion of !MOVABLE blocks would over time make
> > compaction less and less effective too, seeing as it doesn't consider
> > anything !MOVABLE suitable migration targets?
> > 
> 
> It's a solid theory. There has been a lot of activity to weaken fragmentation
> avoidance protection to reduce latency. Unfortunately external fragmentation
> continues to be one of those topics that is very difficult to precisely
> define because it's a matter of definition whether it's important or
> not.

While I generally agree that it's a matter of degree, and a trade-off
between cost and accuracy, what we're observing here is a continued
deterioration of mobility grouping accuracy with uptime, to the point
of over half of memory being in unmovable/reclaimable blocks when the
majority of memory is movable allocations.

The consequences of that are devastating, because actually unmovable
allocations will be grouped less and less efficiently, and that in
turn affects the cost and effectiveness of every compaction run.

> Another avenue worth considering is that compaction used to scan unmovable
> pageblocks and migrate movable pages out of there but that was weakened
> over time trying to allocate THP pages from direct allocation context
> quickly enough. I'm not exactly sure what we do there at the moment and
> whether kcompactd cleans unmovable pageblocks or not. It takes time but
> it also reduces unmovable pageblock steals over time (or at least it did
> a few years ago when I last investigated this in depth).

I don't believe it does. There is a migrate_async_suitable() check
that skips over everything that isn't MOVABLE, but in spite of the
name this check is done for all compaction modes, see:

 isolate_freepages()
  suitable_migration_target()
   migrate_async_suitable()

That's why not even /proc/sys/vm/compact_memory would be able to
defragment these blocks right now.

But the more I think about this issue, the more I think compaction is
the wrong place to address this. Inefficiently packed unmovable blocks
will be less compactable, regardless of how many times the compaction
scanner looks at them. Compaction might be able to get a few chunks in
between actually unmovable pages, but the maximum size of these chunks
will be severely limited, and they won't be able to coalesce with the
surrounding chunks. Compaction cannot really fix up what mobility
grouping lets slide, so reallocating effort from allocation grouping
to compaction scanning will always be a net loss at higher uptimes.

> Unfortunately I do not have any suggestions offhand on how it could be
> easily improved without going back to first principals and identifying
> what pages end up in awkward positions, why and whether the cost of
> "cleaning" unmovable pageblocks during compaction for a high-order
> allocation is justified or not.

I don't think this particular case is a trade-off situation. From 3.10
to current kernels, we have seen both allocation latencies and overall
throughput (number of DB reqs handled per second) get worse.

Thanks!

[toc] | [prev] | [next] | [standalone]

#1493719 — [RFC 2/4] mm, compaction: add migratetype to compact_control

Preparation patch. We are going to need migratetype at lower layers than
compact_zone() and compact_finished().

Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
 mm/compaction.c | 15 +++++++--------
 mm/internal.h   |  1 +
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index 673a81618534..823538353b80 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1290,11 +1290,12 @@ static inline bool is_via_compact_memory(int order)
 	return order == -1;
 }
 
-static enum compact_result __compact_finished(struct zone *zone, struct compact_control *cc,
-			    const int migratetype)
+static enum compact_result __compact_finished(struct zone *zone,
+						struct compact_control *cc)
 {
 	unsigned int order;
 	unsigned long watermark;
+	const int migratetype = cc->migratetype;
 
 	if (cc->contended || fatal_signal_pending(current))
 		return COMPACT_CONTENDED;
@@ -1357,12 +1358,11 @@ static enum compact_result __compact_finished(struct zone *zone, struct compact_
 }
 
 static enum compact_result compact_finished(struct zone *zone,
-			struct compact_control *cc,
-			const int migratetype)
+			struct compact_control *cc)
 {
 	int ret;
 
-	ret = __compact_finished(zone, cc, migratetype);
+	ret = __compact_finished(zone, cc);
 	trace_mm_compaction_finished(zone, cc->order, ret);
 	if (ret == COMPACT_NO_SUITABLE_PAGE)
 		ret = COMPACT_CONTINUE;
@@ -1497,9 +1497,9 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro
 	enum compact_result ret;
 	unsigned long start_pfn = zone->zone_start_pfn;
 	unsigned long end_pfn = zone_end_pfn(zone);
-	const int migratetype = gfpflags_to_migratetype(cc->gfp_mask);
 	const bool sync = cc->mode != MIGRATE_ASYNC;
 
+	cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask);
 	ret = compaction_suitable(zone, cc->order, cc->alloc_flags,
 							cc->classzone_idx);
 	/* Compaction is likely to fail */
@@ -1549,8 +1549,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro
 
 	migrate_prep_local();
 
-	while ((ret = compact_finished(zone, cc, migratetype)) ==
-						COMPACT_CONTINUE) {
+	while ((ret = compact_finished(zone, cc)) == COMPACT_CONTINUE) {
 		int err;
 
 		switch (isolate_migratepages(zone, cc)) {
diff --git a/mm/internal.h b/mm/internal.h
index 537ac9951f5f..1fee63010dcc 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -182,6 +182,7 @@ struct compact_control {
 	bool direct_compaction;		/* False from kcompactd or /proc/... */
 	bool whole_zone;		/* Whole zone should/has been scanned */
 	int order;			/* order a direct compactor needs */
+	int migratetype;		/* migratetype of direct compactor */
 	const gfp_t gfp_mask;		/* gfp mask of a direct compactor */
 	const unsigned int alloc_flags;	/* alloc flags of a direct compactor */
 	const int classzone_idx;	/* zone index of a direct compactor */
-- 
2.10.0

[toc] | [prev] | [next] | [standalone]

#1493721 — [RFC 1/4] mm, compaction: change migrate_async_suitable() to suitable_migration_source()

Preparation for making the decisions more complex and depending on
compact_control flags. No functional change.

Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
 include/linux/mmzone.h |  5 +++++
 mm/compaction.c        | 19 +++++++++++--------
 2 files changed, 16 insertions(+), 8 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 454495cc00fe..9cd3ee58ab2b 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -72,6 +72,11 @@ extern char * const migratetype_names[MIGRATE_TYPES];
 #  define is_migrate_cma(migratetype) false
 #endif
 
+static inline bool is_migrate_movable(int mt)
+{
+	return is_migrate_cma(mt) || mt == MIGRATE_MOVABLE;
+}
+
 #define for_each_migratetype_order(order, type) \
 	for (order = 0; order < MAX_ORDER; order++) \
 		for (type = 0; type < MIGRATE_TYPES; type++)
diff --git a/mm/compaction.c b/mm/compaction.c
index 6e77b2016da1..673a81618534 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -88,11 +88,6 @@ static void map_pages(struct list_head *list)
 	list_splice(&tmp_list, list);
 }
 
-static inline bool migrate_async_suitable(int migratetype)
-{
-	return is_migrate_cma(migratetype) || migratetype == MIGRATE_MOVABLE;
-}
-
 #ifdef CONFIG_COMPACTION
 
 int PageMovable(struct page *page)
@@ -996,6 +991,15 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn,
 #endif /* CONFIG_COMPACTION || CONFIG_CMA */
 #ifdef CONFIG_COMPACTION
 
+static bool suitable_migration_source(struct compact_control *cc,
+							struct page *page)
+{
+	if (cc->mode != MIGRATE_ASYNC)
+		return true;
+
+	return is_migrate_movable(get_pageblock_migratetype(page));
+}
+
 /* Returns true if the page is within a block suitable for migration to */
 static bool suitable_migration_target(struct compact_control *cc,
 							struct page *page)
@@ -1015,7 +1019,7 @@ static bool suitable_migration_target(struct compact_control *cc,
 	}
 
 	/* If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration */
-	if (migrate_async_suitable(get_pageblock_migratetype(page)))
+	if (is_migrate_movable(get_pageblock_migratetype(page)))
 		return true;
 
 	/* Otherwise skip the block */
@@ -1250,8 +1254,7 @@ static isolate_migrate_t isolate_migratepages(struct zone *zone,
 		 * Async compaction is optimistic to see if the minimum amount
 		 * of work satisfies the allocation.
 		 */
-		if (cc->mode == MIGRATE_ASYNC &&
-		    !migrate_async_suitable(get_pageblock_migratetype(page)))
+		if (!suitable_migration_source(cc, page))
 			continue;
 
 		/* Perform the isolation */
-- 
2.10.0

[toc] | [prev] | [next] | [standalone]

#1493723 — [RFC 3/4] mm, compaction: restrict async compaction to matching migratetype

The migrate scanner in async compaction is currently limited to MIGRATE_MOVABLE
pageblocks. This is a heuristic intended to reduce latency, based on the
assumption that non-MOVABLE pageblocks are unlikely to contain movable pages.

However, with the exception of THP's, most high-order allocations are not
movable. Should the async compaction succeed, this increases the chance that
the non-MOVABLE allocations will fallback to a MOVABLE pageblock, making the
long-term fragmentation worse.

This patch attempts to help the situation by changing async direct compaction
so that the migrate scanner only scans the pageblocks of the requested
migratetype. If it's a non-MOVABLE type and there are such pageblocks that do
contain movable pages, chances are that the allocation can succeed within one
of such pageblocks, removing the need for a fallback. If that fails, the
subsequent sync attempt will ignore this restriction.

Not-yet-signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
 mm/compaction.c | 11 +++++++++--
 mm/page_alloc.c | 20 +++++++++++++-------
 2 files changed, 22 insertions(+), 9 deletions(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index 823538353b80..eb4ccd403543 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -994,10 +994,17 @@ isolate_migratepages_range(struct compact_control *cc, unsigned long start_pfn,
 static bool suitable_migration_source(struct compact_control *cc,
 							struct page *page)
 {
-	if (cc->mode != MIGRATE_ASYNC)
+	int block_mt;
+
+	if ((cc->mode != MIGRATE_ASYNC) || !cc->direct_compaction)
 		return true;
 
-	return is_migrate_movable(get_pageblock_migratetype(page));
+	block_mt = get_pageblock_migratetype(page);
+
+	if (cc->migratetype == MIGRATE_MOVABLE)
+		return is_migrate_movable(block_mt);
+	else
+		return block_mt == cc->migratetype;
 }
 
 /* Returns true if the page is within a block suitable for migration to */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index fab8b6913179..0c00beec9336 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3492,6 +3492,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 						struct alloc_context *ac)
 {
 	bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
+	const bool costly_order = order > PAGE_ALLOC_COSTLY_ORDER;
 	struct page *page = NULL;
 	unsigned int alloc_flags;
 	unsigned long did_some_progress;
@@ -3539,12 +3540,17 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 
 	/*
 	 * For costly allocations, try direct compaction first, as it's likely
-	 * that we have enough base pages and don't need to reclaim. Don't try
-	 * that for allocations that are allowed to ignore watermarks, as the
-	 * ALLOC_NO_WATERMARKS attempt didn't yet happen.
+	 * that we have enough base pages and don't need to reclaim. For non-
+	 * movable high-order allocations, do that as well, as compaction will
+	 * try prevent permanent fragmentation by migrating from blocks of the
+	 * same migratetype.
+	 * Don't try this for allocations that are allowed to ignore
+	 * watermarks, as the ALLOC_NO_WATERMARKS attempt didn't yet happen.
 	 */
-	if (can_direct_reclaim && order > PAGE_ALLOC_COSTLY_ORDER &&
-		!gfp_pfmemalloc_allowed(gfp_mask)) {
+	if (can_direct_reclaim &&
+			(costly_order ||
+			   (order > 0 && ac->migratetype != MIGRATE_MOVABLE))
+			&& !gfp_pfmemalloc_allowed(gfp_mask)) {
 		page = __alloc_pages_direct_compact(gfp_mask, order,
 						alloc_flags, ac,
 						INIT_COMPACT_PRIORITY,
@@ -3556,7 +3562,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 		 * Checks for costly allocations with __GFP_NORETRY, which
 		 * includes THP page fault allocations
 		 */
-		if (gfp_mask & __GFP_NORETRY) {
+		if (costly_order && (gfp_mask & __GFP_NORETRY)) {
 			/*
 			 * If compaction is deferred for high-order allocations,
 			 * it is because sync compaction recently failed. If
@@ -3651,7 +3657,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	 * Do not retry costly high order allocations unless they are
 	 * __GFP_REPEAT
 	 */
-	if (order > PAGE_ALLOC_COSTLY_ORDER && !(gfp_mask & __GFP_REPEAT))
+	if (costly_order && !(gfp_mask & __GFP_REPEAT))
 		goto nopage;
 
 	if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,
-- 
2.10.0

[toc] | [prev] | [next] | [standalone]

#1493724 — [RFC 4/4] mm, page_alloc: disallow migratetype fallback in fastpath

The previous patch has adjusted async compaction so that it helps against
longterm fragmentation when compacting for a non-MOVABLE high-order allocation.
The goal of this patch is to force such allocations go through compaction
once before being allowed to fallback to a pageblock of different migratetype
(e.g. MOVABLE). In contexts where compaction is not allowed (and for order-0
allocations), this delayed fallback possibility can still help by trying a
different zone where fallback might not be needed and potentially waking up
kswapd earlier.

Not-yet-signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
 mm/internal.h   |  1 +
 mm/page_alloc.c | 14 ++++++++++----
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 1fee63010dcc..a46eab383e8d 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -466,6 +466,7 @@ unsigned long reclaim_clean_pages_from_list(struct zone *zone,
 #define ALLOC_HIGH		0x20 /* __GFP_HIGH set */
 #define ALLOC_CPUSET		0x40 /* check for correct cpuset */
 #define ALLOC_CMA		0x80 /* allow allocations from CMA areas */
+#define ALLOC_FALLBACK		0x100 /* allow fallback of migratetype */
 
 enum ttu_flags;
 struct tlbflush_unmap_batch;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 0c00beec9336..8a8ef9ebeb4d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2163,7 +2163,7 @@ __rmqueue_fallback(struct zone *zone, unsigned int order, int start_migratetype)
  * Call me with the zone->lock already held.
  */
 static struct page *__rmqueue(struct zone *zone, unsigned int order,
-				int migratetype)
+				int migratetype, bool allow_fallback)
 {
 	struct page *page;
 
@@ -2172,7 +2172,7 @@ static struct page *__rmqueue(struct zone *zone, unsigned int order,
 		if (migratetype == MIGRATE_MOVABLE)
 			page = __rmqueue_cma_fallback(zone, order);
 
-		if (!page)
+		if (!page && allow_fallback)
 			page = __rmqueue_fallback(zone, order, migratetype);
 	}
 
@@ -2193,7 +2193,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,
 
 	spin_lock(&zone->lock);
 	for (i = 0; i < count; ++i) {
-		struct page *page = __rmqueue(zone, order, migratetype);
+		struct page *page = __rmqueue(zone, order, migratetype, true);
 		if (unlikely(page == NULL))
 			break;
 
@@ -2626,7 +2626,10 @@ struct page *buffered_rmqueue(struct zone *preferred_zone,
 					trace_mm_page_alloc_zone_locked(page, order, migratetype);
 			}
 			if (!page)
-				page = __rmqueue(zone, order, migratetype);
+				page = __rmqueue(zone, order, migratetype,
+						alloc_flags &
+						(ALLOC_FALLBACK |
+						 ALLOC_NO_WATERMARKS));
 		} while (page && check_new_pages(page, order));
 		spin_unlock(&zone->lock);
 		if (!page)
@@ -3583,6 +3586,9 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 		}
 	}
 
+	/* async direct compaction didn't help, now allow fallback */
+	alloc_flags |= ALLOC_FALLBACK;
+
 retry:
 	/* Ensure kswapd doesn't accidentally go to sleep as long as we loop */
 	if (gfp_mask & __GFP_KSWAPD_RECLAIM)
-- 
2.10.0

[toc] | [prev] | [next] | [standalone]

#1499772 — Re: [RFC 4/4] mm, page_alloc: disallow migratetype fallback in fastpath

On 09/29/2016 11:05 PM, Vlastimil Babka wrote:
> The previous patch has adjusted async compaction so that it helps against
> longterm fragmentation when compacting for a non-MOVABLE high-order allocation.
> The goal of this patch is to force such allocations go through compaction
> once before being allowed to fallback to a pageblock of different migratetype
> (e.g. MOVABLE). In contexts where compaction is not allowed (and for order-0
> allocations), this delayed fallback possibility can still help by trying a
> different zone where fallback might not be needed and potentially waking up
> kswapd earlier.
> 
> Not-yet-signed-off-by: Vlastimil Babka <vbabka@suse.cz>

I forgot that compaction itself also needs to be told to not allow fallback,
otherwise it finishes with COMPACT_SUCCESS without actually doing anything...

From 93acabcc744eab5a4aa965322e9083d0d9f990fc Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <vbabka@suse.cz>
Date: Wed, 12 Oct 2016 16:36:35 +0200
Subject: fixup! mm, page_alloc: disallow migratetype fallback in fastpath

We want to force compaction to run even when the requested page is potentially
available, but of a wrong migratetype. This won't work unless the compaction
itself is modified to not declare imediatelly success when it sees such page.
---
 mm/compaction.c | 22 +++++++++++++++++-----
 mm/internal.h   |  3 ++-
 2 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/mm/compaction.c b/mm/compaction.c
index eb4ccd403543..eeb9200f7b7e 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1354,9 +1354,11 @@ static enum compact_result __compact_finished(struct zone *zone,
 #endif
 		/*
 		 * Job done if allocation would steal freepages from
-		 * other migratetype buddy lists.
+		 * other migratetype buddy lists. This is not allowed
+		 * for async direct compaction.
 		 */
-		if (find_suitable_fallback(area, order, migratetype,
+		if (!cc->prevent_fallback &&
+			find_suitable_fallback(area, order, migratetype,
 						true, &can_steal) != -1)
 			return COMPACT_SUCCESS;
 	}
@@ -1509,8 +1511,17 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro
 	cc->migratetype = gfpflags_to_migratetype(cc->gfp_mask);
 	ret = compaction_suitable(zone, cc->order, cc->alloc_flags,
 							cc->classzone_idx);
-	/* Compaction is likely to fail */
-	if (ret == COMPACT_SUCCESS || ret == COMPACT_SKIPPED)
+	/*
+	 * Compaction should not be needed. If we don't allow stealing from
+	 * pageblocks of different migratetype, the watermark checks cannot
+	 * distinguish that, so assume we would need to steal, and leave the
+	 * thorough check to compact_finished().
+	 */
+	if (ret == COMPACT_SUCCESS && !cc->prevent_fallback)
+		return ret;
+
+	/* Compaction is likely to fail due to insufficient free pages */
+	if (ret == COMPACT_SKIPPED)
 		return ret;
 
 	/* huh, compaction_suitable is returning something unexpected */
@@ -1678,7 +1689,8 @@ static enum compact_result compact_zone_order(struct zone *zone, int order,
 		.direct_compaction = true,
 		.whole_zone = (prio == MIN_COMPACT_PRIORITY),
 		.ignore_skip_hint = (prio == MIN_COMPACT_PRIORITY),
-		.ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY)
+		.ignore_block_suitable = (prio == MIN_COMPACT_PRIORITY),
+		.prevent_fallback = (prio == COMPACT_PRIO_ASYNC)
 	};
 	INIT_LIST_HEAD(&cc.freepages);
 	INIT_LIST_HEAD(&cc.migratepages);
diff --git a/mm/internal.h b/mm/internal.h
index a46eab383e8d..bb01d9bd60a8 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -181,13 +181,14 @@ struct compact_control {
 	bool ignore_block_suitable;	/* Scan blocks considered unsuitable */
 	bool direct_compaction;		/* False from kcompactd or /proc/... */
 	bool whole_zone;		/* Whole zone should/has been scanned */
+	bool contended;			/* Signal lock or sched contention */
+	bool prevent_fallback;		/* Stealing migratetypes not allowed */
 	int order;			/* order a direct compactor needs */
 	int migratetype;		/* migratetype of direct compactor */
 	const gfp_t gfp_mask;		/* gfp mask of a direct compactor */
 	const unsigned int alloc_flags;	/* alloc flags of a direct compactor */
 	const int classzone_idx;	/* zone index of a direct compactor */
 	struct zone *zone;
-	bool contended;			/* Signal lock or sched contention */
 };
 
 unsigned long
-- 
2.10.0

[toc] | [prev] | [next] | [standalone]

#1500073 — Re: [RFC 4/4] mm, page_alloc: disallow migratetype fallback in fastpath

On Thu, Sep 29, 2016 at 11:05:48PM +0200, Vlastimil Babka wrote:
> The previous patch has adjusted async compaction so that it helps against
> longterm fragmentation when compacting for a non-MOVABLE high-order allocation.
> The goal of this patch is to force such allocations go through compaction
> once before being allowed to fallback to a pageblock of different migratetype
> (e.g. MOVABLE). In contexts where compaction is not allowed (and for order-0
> allocations), this delayed fallback possibility can still help by trying a
> different zone where fallback might not be needed and potentially waking up
> kswapd earlier.

Hmm... can we justify this compaction overhead in case of that there is
high order freepages in other migratetype pageblock? There is no guarantee
that longterm fragmentation happens and it affects the system
peformance.

And, it would easilly fail to compact in unmovable pageblock since
there would not be migratable pages if everything works as our
intended. So, I guess that checking it over and over doesn't help to
reduce fragmentation and just increase latency of allocation.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1500204 — Re: [RFC 4/4] mm, page_alloc: disallow migratetype fallback in fastpath

On 10/13/2016 09:58 AM, Joonsoo Kim wrote:
> On Thu, Sep 29, 2016 at 11:05:48PM +0200, Vlastimil Babka wrote:
>> The previous patch has adjusted async compaction so that it helps against
>> longterm fragmentation when compacting for a non-MOVABLE high-order allocation.
>> The goal of this patch is to force such allocations go through compaction
>> once before being allowed to fallback to a pageblock of different migratetype
>> (e.g. MOVABLE). In contexts where compaction is not allowed (and for order-0
>> allocations), this delayed fallback possibility can still help by trying a
>> different zone where fallback might not be needed and potentially waking up
>> kswapd earlier.
>
> Hmm... can we justify this compaction overhead in case of that there is
> high order freepages in other migratetype pageblock? There is no guarantee
> that longterm fragmentation happens and it affects the system
> peformance.

Yeah, I hoped testing would show whether this makes any difference, and 
what the overhead is, and then we can decide whether it's worth.

> And, it would easilly fail to compact in unmovable pageblock since
> there would not be migratable pages if everything works as our
> intended. So, I guess that checking it over and over doesn't help to
> reduce fragmentation and just increase latency of allocation.

The pageblock isolation_suitable heuristics of compaction should 
mitigate rescanning blocks without success. We could also add a per-zone 
flag that gets set during a fallback allocation event and cleared by 
finished compaction, or something.

> Thanks.
>

[toc] | [prev] | [next] | [standalone]

#1493725 — [RFC 0/4] try to reduce fragmenting fallbacks

Hi Johannes,

here's something quick to try or ponder about. However, untested since it's too
late here. Based on mmotm-2016-09-27-16-08 plus this fix [1]

[1] http://lkml.kernel.org/r/<cadadd38-6456-f58e-504f-cc18ddc47b3f@suse.cz>

Vlastimil Babka (4):
  mm, compaction: change migrate_async_suitable() to
    suitable_migration_source()
  mm, compaction: add migratetype to compact_control
  mm, compaction: restrict async compaction to matching migratetype
  mm, page_alloc: disallow migratetype fallback in fastpath

 include/linux/mmzone.h |  5 +++++
 mm/compaction.c        | 41 +++++++++++++++++++++++++----------------
 mm/internal.h          |  2 ++
 mm/page_alloc.c        | 34 +++++++++++++++++++++++-----------
 4 files changed, 55 insertions(+), 27 deletions(-)

-- 
2.10.0

[toc] | [prev] | [next] | [standalone]

#1497075 — [RFC 5/4] mm, page_alloc: split smallest stolen page in fallback

The __rmqueue_fallback() is called when there's no free page of requested
migratetype, and we need to steal from a different one. There are various
heuristics to make this event infrequent and reduce permanent fragmentation.
The main one is to try stealing from a pageblock that has the most free pages,
and possibly steal them all at once and convert the whole pageblock. Precise
searching for such pageblock would be expensive, so instead the heuristics
walks the free lists from MAX_ORDER down to requested order and assumes that
the block with highest-order free page is likely to also have the most free
pages in total.

So the chances are that together with the highest-order page, we steal also
pages of lower orders from the same block. But then we still split the highest
order page. This is wasteful and can contribute to fragmentation instead of
avoiding it.

This patch thus changes __rmqueue_fallback() to only steal the pages(s) and
put them on a freelist of the requested migratetype, and only report whether
it was successful. Then we pick the smallest page with __rmqueue_smallest().
This is all under zone lock, so nobody can steal it from us in the process.
This should reduce fragmentation due to fallbacks. At worst we are only
stealing a single highest-order page and waste some cycles by moving it between
lists and then removing it, but fallback is not exactly hot path so that should
not be a concern. As a side benefit the patch removes some duplicate code by
reusing __rmqueue_smallest().

Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
 mm/page_alloc.c | 50 ++++++++++++++++++++++++++------------------------
 1 file changed, 26 insertions(+), 24 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 8a8ef9ebeb4d..2ccd80079d22 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1957,14 +1957,24 @@ static bool can_steal_fallback(unsigned int order, int start_mt)
  * use it's pages as requested migratetype in the future.
  */
 static void steal_suitable_fallback(struct zone *zone, struct page *page,
-							  int start_type)
+					 int start_type, bool whole_block)
 {
 	unsigned int current_order = page_order(page);
+	struct free_area *area;
 	int pages;
 
 	/* Take ownership for orders >= pageblock_order */
 	if (current_order >= pageblock_order) {
 		change_pageblock_range(page, current_order, start_type);
+		area = &zone->free_area[current_order];
+		list_move(&page->lru, &area->free_list[start_type]);
+		return;
+	}
+
+	/* We are not allowed to try stealing from the whole block */
+	if (!whole_block) {
+		area = &zone->free_area[current_order];
+		list_move(&page->lru, &area->free_list[start_type]);
 		return;
 	}
 
@@ -2108,8 +2118,13 @@ static void unreserve_highatomic_pageblock(const struct alloc_context *ac)
 	}
 }
 
-/* Remove an element from the buddy allocator from the fallback list */
-static inline struct page *
+/*
+ * Try finding a free buddy page on the fallback list and put it on the free
+ * list of requested migratetype, possibly along with other pages from the same
+ * block, depending on fragmentation avoidance heuristics. Returns true if
+ * fallback was found so that __rmqueue_smallest() can grab it.
+ */
+static inline bool
 __rmqueue_fallback(struct zone *zone, unsigned int order, int start_migratetype)
 {
 	struct free_area *area;
@@ -2130,32 +2145,16 @@ __rmqueue_fallback(struct zone *zone, unsigned int order, int start_migratetype)
 
 		page = list_first_entry(&area->free_list[fallback_mt],
 						struct page, lru);
-		if (can_steal)
-			steal_suitable_fallback(zone, page, start_migratetype);
-
-		/* Remove the page from the freelists */
-		area->nr_free--;
-		list_del(&page->lru);
-		rmv_page_order(page);
 
-		expand(zone, page, order, current_order, area,
-					start_migratetype);
-		/*
-		 * The pcppage_migratetype may differ from pageblock's
-		 * migratetype depending on the decisions in
-		 * find_suitable_fallback(). This is OK as long as it does not
-		 * differ for MIGRATE_CMA pageblocks. Those can be used as
-		 * fallback only via special __rmqueue_cma_fallback() function
-		 */
-		set_pcppage_migratetype(page, start_migratetype);
+		steal_suitable_fallback(zone, page, start_migratetype, can_steal);
 
 		trace_mm_page_alloc_extfrag(page, order, current_order,
 			start_migratetype, fallback_mt);
 
-		return page;
+		return true;
 	}
 
-	return NULL;
+	return false;
 }
 
 /*
@@ -2167,13 +2166,16 @@ static struct page *__rmqueue(struct zone *zone, unsigned int order,
 {
 	struct page *page;
 
+retry:
 	page = __rmqueue_smallest(zone, order, migratetype);
 	if (unlikely(!page)) {
 		if (migratetype == MIGRATE_MOVABLE)
 			page = __rmqueue_cma_fallback(zone, order);
 
-		if (!page && allow_fallback)
-			page = __rmqueue_fallback(zone, order, migratetype);
+		if (!page && allow_fallback) {
+			if (__rmqueue_fallback(zone, order, migratetype))
+				goto retry;
+		}
 	}
 
 	trace_mm_page_alloc_zone_locked(page, order, migratetype);
-- 
2.10.0

[toc] | [prev] | [next] | [standalone]

#1498444 — Re: [RFC 0/4] try to reduce fragmenting fallbacks

Hi Vlastimil,

sorry for the delay, I just got back from traveling.

On Thu, Sep 29, 2016 at 11:05:44PM +0200, Vlastimil Babka wrote:
> Hi Johannes,
> 
> here's something quick to try or ponder about. However, untested since it's too
> late here. Based on mmotm-2016-09-27-16-08 plus this fix [1]
> 
> [1] http://lkml.kernel.org/r/<cadadd38-6456-f58e-504f-cc18ddc47b3f@suse.cz>

Thanks for whipping something up, I'll give these a shot. 4/4 is
something I wondered about too. Let's see how this performs.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web