Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1492231 > unrolled thread
| Started by | Johannes Weiner <hannes@cmpxchg.org> |
|---|---|
| First post | 2016-09-28 03:50 +0200 |
| Last post | 2016-10-13 16:20 +0200 |
| Articles | 2 on this page of 22 — 4 participants |
Back to article view | Back to linux.kernel
Regression in mobility grouping? Johannes Weiner <hannes@cmpxchg.org> - 2016-09-28 03:50 +0200
Re: Regression in mobility grouping? Vlastimil Babka <vbabka@suse.cz> - 2016-09-28 11:10 +0200
Re: Regression in mobility grouping? Johannes Weiner <hannes@cmpxchg.org> - 2016-09-28 17:50 +0200
Re: Regression in mobility grouping? Johannes Weiner <hannes@cmpxchg.org> - 2016-09-29 04:30 +0200
Re: Regression in mobility grouping? Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2016-09-29 08:10 +0200
Re: Regression in mobility grouping? Johannes Weiner <hannes@cmpxchg.org> - 2016-09-29 18:20 +0200
Re: Regression in mobility grouping? Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2016-10-13 09:40 +0200
Re: Regression in mobility grouping? Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 09:20 +0200
Re: Regression in mobility grouping? Mel Gorman <mgorman@suse.de> - 2016-09-28 12:30 +0200
Re: Regression in mobility grouping? Johannes Weiner <hannes@cmpxchg.org> - 2016-09-28 18:50 +0200
[RFC 2/4] mm, compaction: add migratetype to compact_control Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 23:10 +0200
[RFC 1/4] mm, compaction: change migrate_async_suitable() to suitable_migration_source() Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 23:10 +0200
[RFC 3/4] mm, compaction: restrict async compaction to matching migratetype Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 23:10 +0200
[RFC 4/4] mm, page_alloc: disallow migratetype fallback in fastpath Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 23:10 +0200
Re: [RFC 4/4] mm, page_alloc: disallow migratetype fallback in fastpath Vlastimil Babka <vbabka@suse.cz> - 2016-10-12 17:20 +0200
Re: [RFC 4/4] mm, page_alloc: disallow migratetype fallback in fastpath Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2016-10-13 10:00 +0200
Re: [RFC 4/4] mm, page_alloc: disallow migratetype fallback in fastpath Vlastimil Babka <vbabka@suse.cz> - 2016-10-13 13:50 +0200
[RFC 0/4] try to reduce fragmenting fallbacks Vlastimil Babka <vbabka@suse.cz> - 2016-09-29 23:10 +0200
[RFC 5/4] mm, page_alloc: split smallest stolen page in fallback Vlastimil Babka <vbabka@suse.cz> - 2016-10-07 10:40 +0200
Re: [RFC 0/4] try to reduce fragmenting fallbacks Johannes Weiner <hannes@cmpxchg.org> - 2016-10-10 19:30 +0200
[RFC 6/4] mm, page_alloc: introduce MIGRATE_MIXED migratetype Vlastimil Babka <vbabka@suse.cz> - 2016-10-11 15:20 +0200
[RFC 7/4] mm, page_alloc: count movable pages when stealing Vlastimil Babka <vbabka@suse.cz> - 2016-10-13 16:20 +0200
Page 2 of 2 — ← Prev page 1 [2]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-10-11 15:20 +0200 |
| Subject | [RFC 6/4] mm, page_alloc: introduce MIGRATE_MIXED migratetype |
| Message-ID | <sr58l-7aD-9@gated-at.bofh.it> |
| In reply to | #1493725 |
Page mobility grouping tries to minimize the number of pageblocks that contain
non-migratable pages by distinguishing MOVABLE, UNMOVABLE and RECLAIMABLE
pageblock migratetypes. Changing pageblock's migratetype is allowed if an
allocation of different migratetype steals more than half of pages from it.
That means it's possible to have pageblocks that contain some UNMOVABLE and
RECLAIMABLE pages, yet they are marked as MOVABLE, and the next time stealing
happens, another MOVABLE pageblock might get polluted. On the other hand, if we
duly marked all polluted pageblocks (even just by single page) as UNMOVABLE or
RECLAIMABLE, further allocations and freeing of pages would tend to spread over
all of them, and there would be little pressure for them to eventually become
fully free and MOVABLE.
This patch thus introduces a new migratetype MIGRATE_MIXED, which is intended
to mark pageblocks that contain some UNMOVABLE or RECLAIMABLE pages, but not
enough to mark the whole pageblocks as such. These pageblocks become preferred
fallback before UNMOVABLE/RECLAIMABLE allocation steals from a MOVABLE
pageblock, or vice versa. This should help page mobility grouping:
- UNMOVABLE and RECLAIMABLE allocations will try to be satisfied from their
respective pageblocks. If these are full, polluting other pageblocks is
limited to MIGRATE_MIXED pageblocks. MIGRATE_MOVABLE pageblocks remain pure.
If a temporery pressure for UNMOVABLE and RECLAIMABLE pageblocks disappears
and can be satisfied without fallback, the MIXED pageblocks might eventually
fully recover from the polluted pages.
- MOVABLE allocations will exhaust MOVABLE pageblocks first, then fallback to
MIXED as second. This leaves free pages in UNMOVABLE and RECLAIMABLE
pageblocks as a last resort, so those allocations don't have to fall back
so much.
Not-yet-signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
This is not a new idea, but maybe it's time to give it a shot. Perhaps it will
have to be complemented by compaction migrate scanner recovering pageblocks
from one migratetype to another depending on how many free pages and migratable
pages it finds in them. The fallback events have limited information and
unpredictable timing.
include/linux/mmzone.h | 1 +
mm/compaction.c | 14 +++++++++--
mm/page_alloc.c | 63 ++++++++++++++++++++++++++++++++++++--------------
3 files changed, 59 insertions(+), 19 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 9cd3ee58ab2b..e4e0a1f64801 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -41,6 +41,7 @@ enum {
MIGRATE_RECLAIMABLE,
MIGRATE_PCPTYPES, /* the number of types on the pcp lists */
MIGRATE_HIGHATOMIC = MIGRATE_PCPTYPES,
+ MIGRATE_MIXED,
#ifdef CONFIG_CMA
/*
* MIGRATE_CMA migration type is designed to mimic the way
diff --git a/mm/compaction.c b/mm/compaction.c
index eb4ccd403543..79bff09a5cac 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1001,6 +1001,9 @@ static bool suitable_migration_source(struct compact_control *cc,
block_mt = get_pageblock_migratetype(page);
+ if (block_mt == MIGRATE_MIXED)
+ return true;
+
if (cc->migratetype == MIGRATE_MOVABLE)
return is_migrate_movable(block_mt);
else
@@ -1011,6 +1014,8 @@ static bool suitable_migration_source(struct compact_control *cc,
static bool suitable_migration_target(struct compact_control *cc,
struct page *page)
{
+ int block_mt;
+
if (cc->ignore_block_suitable)
return true;
@@ -1025,8 +1030,13 @@ static bool suitable_migration_target(struct compact_control *cc,
return false;
}
- /* If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration */
- if (is_migrate_movable(get_pageblock_migratetype(page)))
+ block_mt = get_pageblock_migratetype(page);
+
+ /*
+ * If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration.
+ * Allow also mixed pageblocks so we are not so restrictive.
+ */
+ if (is_migrate_movable(block_mt) || block_mt == MIGRATE_MIXED)
return true;
/* Otherwise skip the block */
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 2ccd80079d22..6c5bc6a7858c 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -229,6 +229,7 @@ char * const migratetype_names[MIGRATE_TYPES] = {
"Movable",
"Reclaimable",
"HighAtomic",
+ "Mixed",
#ifdef CONFIG_CMA
"CMA",
#endif
@@ -1814,9 +1815,9 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order,
* the free lists for the desirable migrate type are depleted
*/
static int fallbacks[MIGRATE_TYPES][4] = {
- [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MOVABLE, MIGRATE_TYPES },
- [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MOVABLE, MIGRATE_TYPES },
- [MIGRATE_MOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES },
+ [MIGRATE_UNMOVABLE] = { MIGRATE_RECLAIMABLE, MIGRATE_MIXED, MIGRATE_MOVABLE, MIGRATE_TYPES },
+ [MIGRATE_RECLAIMABLE] = { MIGRATE_UNMOVABLE, MIGRATE_MIXED, MIGRATE_MOVABLE, MIGRATE_TYPES },
+ [MIGRATE_MOVABLE] = { MIGRATE_MIXED, MIGRATE_RECLAIMABLE, MIGRATE_UNMOVABLE, MIGRATE_TYPES },
#ifdef CONFIG_CMA
[MIGRATE_CMA] = { MIGRATE_TYPES }, /* Never used */
#endif
@@ -1937,13 +1938,12 @@ static bool can_steal_fallback(unsigned int order, int start_mt)
* but, below check doesn't guarantee it and that is just heuristic
* so could be changed anytime.
*/
- if (order >= pageblock_order)
+ if (order >= pageblock_order || page_group_by_mobility_disabled)
return true;
if (order >= pageblock_order / 2 ||
start_mt == MIGRATE_RECLAIMABLE ||
- start_mt == MIGRATE_UNMOVABLE ||
- page_group_by_mobility_disabled)
+ start_mt == MIGRATE_UNMOVABLE)
return true;
return false;
@@ -1962,6 +1962,7 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page,
unsigned int current_order = page_order(page);
struct free_area *area;
int pages;
+ int old_block_type, new_block_type;
/* Take ownership for orders >= pageblock_order */
if (current_order >= pageblock_order) {
@@ -1975,15 +1976,40 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page,
if (!whole_block) {
area = &zone->free_area[current_order];
list_move(&page->lru, &area->free_list[start_type]);
- return;
+ pages = 1 << current_order;
+ } else {
+ pages = move_freepages_block(zone, page, start_type);
}
- pages = move_freepages_block(zone, page, start_type);
+ new_block_type = old_block_type = get_pageblock_migratetype(page);
+ if (page_group_by_mobility_disabled)
+ new_block_type = start_type;
- /* Claim the whole block if over half of it is free */
- if (pages >= (1 << (pageblock_order-1)) ||
- page_group_by_mobility_disabled)
- set_pageblock_migratetype(page, start_type);
+ if (pages >= (1 << (pageblock_order-1))) {
+ /*
+ * Claim the whole block if over half of it is free. The
+ * exception is the transition to MIGRATE_MOVABLE where we
+ * require it to be fully free so that MIGRATE_MOVABLE
+ * pageblocks consist of purely movable pages. So if we steal
+ * less than whole pageblock, mark it as MIGRATE_MIXED.
+ */
+ if (start_type == MIGRATE_MOVABLE)
+ new_block_type = MIGRATE_MIXED;
+ else
+ new_block_type = start_type;
+ } else {
+ /*
+ * We didn't steal enough to change the block's migratetype.
+ * But if we are stealing from a MOVABLE block for a
+ * non-MOVABLE allocation, mark the block as MIXED.
+ */
+ if (old_block_type == MIGRATE_MOVABLE
+ && start_type != MIGRATE_MOVABLE)
+ new_block_type = MIGRATE_MIXED;
+ }
+
+ if (new_block_type != old_block_type)
+ set_pageblock_migratetype(page, new_block_type);
}
/*
@@ -2526,16 +2552,18 @@ int __isolate_free_page(struct page *page, unsigned int order)
rmv_page_order(page);
/*
- * Set the pageblock if the isolated page is at least half of a
- * pageblock
+ * Set the pageblock's migratetype to MIXED if the isolated page is
+ * at least half of a pageblock, MOVABLE if at least whole pageblock
*/
if (order >= pageblock_order - 1) {
struct page *endpage = page + (1 << order) - 1;
+ int new_mt = (order >= pageblock_order) ?
+ MIGRATE_MOVABLE : MIGRATE_MIXED;
for (; page < endpage; page += pageblock_nr_pages) {
int mt = get_pageblock_migratetype(page);
- if (!is_migrate_isolate(mt) && !is_migrate_cma(mt))
- set_pageblock_migratetype(page,
- MIGRATE_MOVABLE);
+
+ if (!is_migrate_isolate(mt) && !is_migrate_movable(mt))
+ set_pageblock_migratetype(page, new_mt);
}
}
@@ -4213,6 +4241,7 @@ static void show_migration_types(unsigned char type)
[MIGRATE_MOVABLE] = 'M',
[MIGRATE_RECLAIMABLE] = 'E',
[MIGRATE_HIGHATOMIC] = 'H',
+ [MIGRATE_MIXED] = 'M',
#ifdef CONFIG_CMA
[MIGRATE_CMA] = 'C',
#endif
--
2.10.0
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-10-13 16:20 +0200 |
| Subject | [RFC 7/4] mm, page_alloc: count movable pages when stealing |
| Message-ID | <srP1v-4Cs-17@gated-at.bofh.it> |
| In reply to | #1493725 |
When stealing pages from pageblock of a different migratetype, we count how
many free pages were stolen, and change the pageblock's migratetype if more
than half of the pageblock was free. This might be too conservative, as there
might be other pages that are not free, but were allocated with the same
migratetype as our allocation requested.
While we cannot determine the migratetype of allocated pages precisely (at
least without the page_owner functionality enabled), we can count pages that
compaction would try to isolate for migration - those are either on LRU or
__PageMovable(). The rest can be assumed to be MIGRATE_RECLAIMABLE or
MIGRATE_UNMOVABLE, which we cannot easily distinguish. This counting can be
done as part of free page stealing with little additional overhead.
The page stealing code is changed so that it considers free pages plus pages
of the "good" migratetype for the decision whether to change pageblock's
migratetype. For changing pageblock to MIGRATE_MOVABLE, we require that all
pages are either free or appear to be movable, otherwise we use MIGRATE_MIXED.
The result should be more accurate migratetype of pageblocks wrt the actual
pages in the pageblocks, when stealing from semi-occupied pageblocks. This
should help with page grouping by mobility.
Not-yet-signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
include/linux/page-isolation.h | 5 +--
mm/page_alloc.c | 73 +++++++++++++++++++++++++++++-------------
mm/page_isolation.c | 5 +--
3 files changed, 54 insertions(+), 29 deletions(-)
diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h
index 047d64706f2a..d4cd2014fa6f 100644
--- a/include/linux/page-isolation.h
+++ b/include/linux/page-isolation.h
@@ -33,10 +33,7 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count,
bool skip_hwpoisoned_pages);
void set_pageblock_migratetype(struct page *page, int migratetype);
int move_freepages_block(struct zone *zone, struct page *page,
- int migratetype);
-int move_freepages(struct zone *zone,
- struct page *start_page, struct page *end_page,
- int migratetype);
+ int migratetype, int *num_movable);
/*
* Changes migrate type in [start_pfn, end_pfn) to be MIGRATE_ISOLATE.
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 6c5bc6a7858c..29e44364a02d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1842,9 +1842,9 @@ static inline struct page *__rmqueue_cma_fallback(struct zone *zone,
* Note that start_page and end_pages are not aligned on a pageblock
* boundary. If alignment is required, use move_freepages_block()
*/
-int move_freepages(struct zone *zone,
+static int move_freepages(struct zone *zone,
struct page *start_page, struct page *end_page,
- int migratetype)
+ int migratetype, int *num_movable)
{
struct page *page;
unsigned int order;
@@ -1861,6 +1861,9 @@ int move_freepages(struct zone *zone,
VM_BUG_ON(page_zone(start_page) != page_zone(end_page));
#endif
+ if (num_movable)
+ *num_movable = 0;
+
for (page = start_page; page <= end_page;) {
/* Make sure we are not inadvertently changing nodes */
VM_BUG_ON_PAGE(page_to_nid(page) != zone_to_nid(zone), page);
@@ -1870,23 +1873,33 @@ int move_freepages(struct zone *zone,
continue;
}
- if (!PageBuddy(page)) {
- page++;
+ if (PageBuddy(page)) {
+ order = page_order(page);
+ list_move(&page->lru,
+ &zone->free_area[order].free_list[migratetype]);
+ page += 1 << order;
+ pages_moved += 1 << order;
continue;
}
- order = page_order(page);
- list_move(&page->lru,
- &zone->free_area[order].free_list[migratetype]);
- page += 1 << order;
- pages_moved += 1 << order;
+ page++;
+ if (!num_movable)
+ continue;
+
+ /*
+ * We assume that pages that could be isolated for migration are
+ * movable. But we don't actually try isolating, as that would be
+ * expensive.
+ */
+ if (PageLRU(page) || __PageMovable(page))
+ (*num_movable)++;
}
return pages_moved;
}
int move_freepages_block(struct zone *zone, struct page *page,
- int migratetype)
+ int migratetype, int *num_movable)
{
unsigned long start_pfn, end_pfn;
struct page *start_page, *end_page;
@@ -1903,7 +1916,8 @@ int move_freepages_block(struct zone *zone, struct page *page,
if (!zone_spans_pfn(zone, end_pfn))
return 0;
- return move_freepages(zone, start_page, end_page, migratetype);
+ return move_freepages(zone, start_page, end_page, migratetype,
+ num_movable);
}
static void change_pageblock_range(struct page *pageblock_page,
@@ -1961,7 +1975,7 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page,
{
unsigned int current_order = page_order(page);
struct free_area *area;
- int pages;
+ int free_pages, good_pages;
int old_block_type, new_block_type;
/* Take ownership for orders >= pageblock_order */
@@ -1976,24 +1990,37 @@ static void steal_suitable_fallback(struct zone *zone, struct page *page,
if (!whole_block) {
area = &zone->free_area[current_order];
list_move(&page->lru, &area->free_list[start_type]);
- pages = 1 << current_order;
+ free_pages = 1 << current_order;
+ /* We didn't scan the block, so be pessimistic */
+ good_pages = 0;
} else {
- pages = move_freepages_block(zone, page, start_type);
+ free_pages = move_freepages_block(zone, page, start_type,
+ &good_pages);
+ /*
+ * good_pages is now the number of movable pages, but if we
+ * want MOVABLE or RECLAIMABLE, we consider all non-movable as
+ * good (but we can't fully distinguish them)
+ */
+ if (start_type != MIGRATE_MOVABLE)
+ good_pages = pageblock_nr_pages - free_pages -
+ good_pages;
}
new_block_type = old_block_type = get_pageblock_migratetype(page);
if (page_group_by_mobility_disabled)
new_block_type = start_type;
- if (pages >= (1 << (pageblock_order-1))) {
+ if (free_pages + good_pages >= (1 << (pageblock_order-1))) {
/*
- * Claim the whole block if over half of it is free. The
- * exception is the transition to MIGRATE_MOVABLE where we
- * require it to be fully free so that MIGRATE_MOVABLE
- * pageblocks consist of purely movable pages. So if we steal
- * less than whole pageblock, mark it as MIGRATE_MIXED.
+ * Claim the whole block if over half of it is free or of a good
+ * type. The exception is the transition to MIGRATE_MOVABLE
+ * where we require it to be fully free/good so that
+ * MIGRATE_MOVABLE pageblocks consist of purely movable pages.
+ * So if we steal less than whole pageblock, mark it as
+ * MIGRATE_MIXED.
*/
- if (start_type == MIGRATE_MOVABLE)
+ if ((start_type == MIGRATE_MOVABLE) &&
+ free_pages + good_pages < pageblock_nr_pages)
new_block_type = MIGRATE_MIXED;
else
new_block_type = start_type;
@@ -2079,7 +2106,7 @@ static void reserve_highatomic_pageblock(struct page *page, struct zone *zone,
!is_migrate_isolate(mt) && !is_migrate_cma(mt)) {
zone->nr_reserved_highatomic += pageblock_nr_pages;
set_pageblock_migratetype(page, MIGRATE_HIGHATOMIC);
- move_freepages_block(zone, page, MIGRATE_HIGHATOMIC);
+ move_freepages_block(zone, page, MIGRATE_HIGHATOMIC, NULL);
}
out_unlock:
@@ -2136,7 +2163,7 @@ static void unreserve_highatomic_pageblock(const struct alloc_context *ac)
* may increase.
*/
set_pageblock_migratetype(page, ac->migratetype);
- move_freepages_block(zone, page, ac->migratetype);
+ move_freepages_block(zone, page, ac->migratetype, NULL);
spin_unlock_irqrestore(&zone->lock, flags);
return;
}
diff --git a/mm/page_isolation.c b/mm/page_isolation.c
index a5594bfcc5ed..29c2f9b9aba7 100644
--- a/mm/page_isolation.c
+++ b/mm/page_isolation.c
@@ -66,7 +66,8 @@ static int set_migratetype_isolate(struct page *page,
set_pageblock_migratetype(page, MIGRATE_ISOLATE);
zone->nr_isolate_pageblock++;
- nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE);
+ nr_pages = move_freepages_block(zone, page, MIGRATE_ISOLATE,
+ NULL);
__mod_zone_freepage_state(zone, -nr_pages, migratetype);
}
@@ -120,7 +121,7 @@ static void unset_migratetype_isolate(struct page *page, unsigned migratetype)
* pageblock scanning for freepage moving.
*/
if (!isolated_page) {
- nr_pages = move_freepages_block(zone, page, migratetype);
+ nr_pages = move_freepages_block(zone, page, migratetype, NULL);
__mod_zone_freepage_state(zone, nr_pages, migratetype);
}
set_pageblock_migratetype(page, migratetype);
--
2.10.0
[toc] | [prev] | [standalone]
Page 2 of 2 — ← Prev page 1 [2]
Back to top | Article view | linux.kernel
csiph-web