Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1171782 > unrolled thread
| Started by | Joonsoo Kim <iamjoonsoo.kim@lge.com> |
|---|---|
| First post | 2015-06-25 02:50 +0200 |
| Last post | 2015-06-26 04:20 +0200 |
| Articles | 9 — 3 participants |
Back to article view | Back to linux.kernel
[RFC PATCH 00/10] redesign compaction algorithm Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2015-06-25 02:50 +0200
[RFC PATCH 04/10] mm/compaction: clean-up restarting condition check Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2015-06-25 02:50 +0200
[RFC PATCH 08/10] mm/compaction: remove compaction deferring Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2015-06-25 02:50 +0200
[RFC PATCH 03/10] mm/compaction: always update cached pfn Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2015-06-25 02:50 +0200
Re: [RFC PATCH 03/10] mm/compaction: always update cached pfn Vlastimil Babka <vbabka@suse.cz> - 2015-06-25 11:10 +0200
[RFC PATCH 05/10] mm/compaction: make freepage scanner scans non-movable pageblock Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2015-06-25 02:50 +0200
[RFC PATCH 01/10] mm/compaction: update skip-bit if whole pageblock is really scanned Joonsoo Kim <iamjoonsoo.kim@lge.com> - 2015-06-25 02:50 +0200
Re: [RFC PATCH 00/10] redesign compaction algorithm Joonsoo Kim <js1304@gmail.com> - 2015-06-26 04:10 +0200
Re: [RFC PATCH 00/10] redesign compaction algorithm Joonsoo Kim <js1304@gmail.com> - 2015-06-26 04:20 +0200
| From | Joonsoo Kim <iamjoonsoo.kim@lge.com> |
|---|---|
| Date | 2015-06-25 02:50 +0200 |
| Subject | [RFC PATCH 00/10] redesign compaction algorithm |
| Message-ID | <pF3wB-W9-3@gated-at.bofh.it> |
Recently, I got a report that android get slow due to order-2 page
allocation. With some investigation, I found that compaction usually
fails and many pages are reclaimed to make order-2 freepage. I can't
analyze detailed reason that causes compaction fail because I don't
have reproducible environment and compaction code is changed so much
from that version, v3.10. But, I was inspired by this report and started
to think limitation of current compaction algorithm.
Limitation of current compaction algorithm:
1) Migrate scanner can't scan behind of free scanner, because
each scanner starts at both side of zone and go toward each other. If
they meet at some point, compaction is stopped and scanners' position
is reset to both side of zone again. From my experience, migrate scanner
usually doesn't scan beyond of half of the zone range.
2) Compaction capability is highly depends on amount of free memory.
If there is 50 MB free memory on 4 GB system, migrate scanner can
migrate 50 MB used pages at maximum and then will meet free scanner.
If compaction can't make enough high order freepages during this
amount of work, compaction would fail. There is no way to escape this
failure situation in current algorithm and it will scan same region and
fail again and again. And then, it goes into compaction deferring logic
and will be deferred for some times.
3) Compaction capability is highly depends on migratetype of memory,
because freepage scanner doesn't scan unmovable pageblock.
To investigate compaction limitations, I made some compaction benchmarks.
Base environment of this benchmark is fragmented memory. Before testing,
25% of total size of memory is allocated. With some tricks, these
allocations are evenly distributed to whole memory range. So, after
allocation is finished, memory is highly fragmented and possibility of
successful order-3 allocation is very low. Roughly 1500 order-3 allocation
can be successful. Tests attempt excessive amount of allocation request,
that is, 3000, to find out algorithm limitation.
There are two variations.
pageblock type (unmovable / movable):
One is that most pageblocks are unmovable migratetype and the other is
that most pageblocks are movable migratetype.
memory usage (memory hogger 200 MB / kernel build with -j8):
Memory hogger means that 200 MB free memory is occupied by hogger.
Kernel build means that kernel build is running on background and it
will consume free memory, but, amount of consumption will be very
fluctuated.
With these variations, I made 4 test cases by mixing them.
hogger-frag-unmovable
hogger-frag-movable
build-frag-unmovable
build-frag-movable
All tests are conducted on 512 MB QEMU virtual machine with 8 CPUs.
I can easily check weakness of compaction algorithm by following test.
To check 1), hogger-frag-movable benchmark is used. Result is as
following.
bzImage-improve-base
compact_free_scanned 5240676
compact_isolated 75048
compact_migrate_scanned 2468387
compact_stall 710
compact_success 98
pgmigrate_success 34869
Success: 25
Success(N): 53
Column 'Success' and 'Success(N) are calculated by following equations.
Success = successful allocation * 100 / attempts
Success(N) = successful allocation * 100 /
number of successful order-3 allocation
As mentioned above, there are roughly 1500 high order page candidates,
but, compaction just returns 53% of them. With new compaction approach,
it can be increased to 94%. See result at the end of this cover-letter.
To check 2), hogger-frag-movable benchmark is used again, but, with some
tweaks. Amount of allocated memory by memory hogger varys.
bzImage-improve-base
Hogger: 150MB 200MB 250MB 300MB
Success: 41 25 17 9
Success(N): 87 53 37 22
As background knowledge, up to 250MB, there is enough
memory to succeed all order-3 allocation attempts. In 300MB case,
available memory before starting allocation attempt is just 57MB,
so all of attempts cannot succeed.
Anyway, as free memory decreases, compaction success rate also decreases.
It is better to remove this dependency to get stable compaction result
in any case.
To check 3), build-frag-unmovable/movable benchmarks are used.
All factors are same except pageblock migratetypes.
Test: build-frag-unmovable
bzImage-improve-base
compact_free_scanned 5032378
compact_isolated 53368
compact_migrate_scanned 1456516
compact_stall 538
compact_success 93
pgmigrate_success 19926
Success: 15
Success(N): 33
Test: build-frag-movable
bzImage-improve-base
compact_free_scanned 3059086
compact_isolated 129085
compact_migrate_scanned 5029856
compact_stall 388
compact_success 99
pgmigrate_success 52898
Success: 38
Success(N): 82
Pageblock migratetype makes big difference on success rate. 3) would be
one of reason related to this result. Because freepage scanner doesn't
scan non-movable pageblock, compaction can't get enough freepage for
migration and compaction easily fails. This patchset try to solve it
by allowing freepage scanner to scan on non-movable pageblock.
Result show that we cannot get all possible high order page through
current compaction algorithm. And, in case that migratetype of
pageblock is unmovable, success rate get worse. Although we can solve
problem 3) in current algorithm, there is unsolvable limitations, 1), 2),
so I'd like to change compaction algorithm.
This patchset try to solve these limitations by introducing new compaction
approach. Main changes of this patchset are as following:
1) Make freepage scanner scans non-movable pageblock
Watermark check doesn't consider how many pages in non-movable pageblock.
To fully utilize existing freepage, freepage scanner should scan
non-movable pageblock.
2) Introduce compaction depletion state
Compaction algorithm will be changed to scan whole zone range. In this
approach, compaction inevitably do back and forth migration between
different iterations. If back and forth migration can make highorder
freepage, it can be justified. But, in case of depletion of compaction
possiblity, this back and forth migration causes unnecessary overhead.
Compaction depleteion state is introduced to avoid this useless
back and forth migration by detecting depletion of compaction possibility.
3) Change scanner's behaviour
Migration scanner is changed to scan whole zone range regardless freepage
scanner position. Freepage scanner also scans whole zone from
zone_start_pfn to zone_end_pfn. To prevent back and forth migration
within one compaction iteration, freepage scanner marks skip-bit when
scanning pageblock. Migration scanner will skip this marked pageblock.
Finish condition is very simple. If migration scanner reaches end of
the zone, compaction will be finished. If freepage scanner reaches end of
the zone first, it restart at zone_start_pfn. This helps us to overcome
dependency on amount of free memory.
Following is all test results of this patchset.
Test: hogger-frag-unmovable
base nonmovable redesign threshold
compact_free_scanned 2800710 5615427 6441095 2235764
compact_isolated 58323 114183 2711081 647701
compact_migrate_scanned 1078970 2437597 4175464 1697292
compact_stall 341 1066 2059 2092
compact_success 80 123 207 210
pgmigrate_success 27034 53832 1348113 318395
Success: 22 29 44 40
Success(N): 46 61 90 83
Test: hogger-frag-movable
base nonmovable redesign threshold
compact_free_scanned 5240676 5883401 8103231 1860428
compact_isolated 75048 83201 3108978 427602
compact_migrate_scanned 2468387 2755690 4316163 1474287
compact_stall 710 664 2117 1964
compact_success 98 102 234 183
pgmigrate_success 34869 38663 1547318 208629
Success: 25 26 45 44
Success(N): 53 56 94 92
Test: build-frag-unmovable
base nonmovable redesign threshold
compact_free_scanned 5032378 4110920 2538420 1891170
compact_isolated 53368 330762 1020908 534680
compact_migrate_scanned 1456516 6164677 4809150 2667823
compact_stall 538 746 2609 2500
compact_success 93 350 438 403
pgmigrate_success 19926 152754 491609 251977
Success: 15 31 39 40
Success(N): 33 65 80 81
Test: build-frag-movable
base nonmovable redesign threshold
compact_free_scanned 3059086 3852269 2359553 1461131
compact_isolated 129085 238856 907515 387373
compact_migrate_scanned 5029856 5051868 3785605 2177090
compact_stall 388 540 2195 2157
compact_success 99 218 247 225
pgmigrate_success 52898 110021 439739 182366
Success: 38 37 43 43
Success(N): 82 77 89 90
Test: hogger-frag-movable with free memory variation
Hogger: 150MB 200MB 250MB 300MB
bzImage-improve-base
Success: 41 25 17 9
Success(N): 87 53 37 22
bzImage-improve-threshold
Success: 44 44 42 37
Success(N): 94 92 91 80
Test: stress-highalloc in mmtests
(tweaks to request order-7 unmovable allocation)
Ops 1 30.00 8.33 84.67 78.00
Ops 2 32.33 26.67 84.33 79.00
Ops 3 91.67 92.00 95.00 94.00
Compaction stalls 5110 5581 10296 10475
Compaction success 1787 1807 5173 4744
Compaction failures 3323 3774 5123 5731
Compaction pages isolated 6370911 15421622 30534650 11825921
Compaction migrate scanned 52681405 83721428 150444732 53517273
Compaction free scanned 418049611 579768237 310629538 139433577
Compaction cost 3745 8822 17324 6628
Result shows that much improvement comes from redesign algorithm but it
causes too much overhead. However, further optimization reduces this
overhead greatly with a little success rate degradation.
We can observe regression from a patch that allows scanning on
non-movable pageblock in some cases. Although this regression is bad,
there are also much improvement in other cases when most of pageblocks
are non-movable migratetype. IMHO, that patch can be justified by
improvement case. Moreover, this regression disappears after applying
following patches so we don't need to worry.
Please see result of "hogger-frag-movable with free memory variation".
It shows that patched version solves limitations of current compaction
algorithm and almost possible order-3 candidates can be allocated
regardless of amount of free memory.
This patchset is based on next-20150515.
Feel free to comment. :)
Thanks.
Joonsoo Kim (10):
mm/compaction: update skip-bit if whole pageblock is really scanned
mm/compaction: skip useless pfn for scanner's cached pfn
mm/compaction: always update cached pfn
mm/compaction: clean-up restarting condition check
mm/compaction: make freepage scanner scans non-movable pageblock
mm/compaction: introduce compaction depleted state on zone
mm/compaction: limit compaction activity in compaction depleted state
mm/compaction: remove compaction deferring
mm/compaction: redesign compaction
mm/compaction: new threshold for compaction depleted zone
include/linux/compaction.h | 14 +-
include/linux/mmzone.h | 6 +-
include/trace/events/compaction.h | 30 ++--
mm/compaction.c | 353 ++++++++++++++++++++++----------------
mm/internal.h | 1 +
mm/page_alloc.c | 2 +-
mm/vmscan.c | 4 +-
7 files changed, 229 insertions(+), 181 deletions(-)
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [next] | [standalone]
| From | Joonsoo Kim <iamjoonsoo.kim@lge.com> |
|---|---|
| Date | 2015-06-25 02:50 +0200 |
| Subject | [RFC PATCH 04/10] mm/compaction: clean-up restarting condition check |
| Message-ID | <pF3wB-W9-15@gated-at.bofh.it> |
| In reply to | #1171782 |
Rename check function and move one outer condition check to this function.
There is no functional change.
Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
---
mm/compaction.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index 2d8e211..dd2063b 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -188,8 +188,11 @@ void compaction_defer_reset(struct zone *zone, int order,
}
/* Returns true if restarting compaction after many failures */
-bool compaction_restarting(struct zone *zone, int order)
+static bool compaction_direct_restarting(struct zone *zone, int order)
{
+ if (current_is_kswapd())
+ return false;
+
if (order < zone->compact_order_failed)
return false;
@@ -1327,7 +1330,7 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
* is about to be retried after being deferred. kswapd does not do
* this reset as it'll reset the cached information when going to sleep.
*/
- if (compaction_restarting(zone, cc->order) && !current_is_kswapd())
+ if (compaction_direct_restarting(zone, cc->order))
__reset_isolation_suitable(zone);
/*
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Joonsoo Kim <iamjoonsoo.kim@lge.com> |
|---|---|
| Date | 2015-06-25 02:50 +0200 |
| Subject | [RFC PATCH 08/10] mm/compaction: remove compaction deferring |
| Message-ID | <pF3wB-W9-17@gated-at.bofh.it> |
| In reply to | #1171782 |
Now, we have a way to determine compaction depleted state and compaction
activity will be limited according this state and depletion depth so
compaction overhead would be well controlled without compaction deferring.
So, this patch remove compaction deferring completely.
Various functions are renamed and tracepoint outputs are changed due to
this removing.
Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
---
include/linux/compaction.h | 14 +-------
include/linux/mmzone.h | 3 +-
include/trace/events/compaction.h | 30 +++++++---------
mm/compaction.c | 74 ++++++++++-----------------------------
mm/page_alloc.c | 2 +-
mm/vmscan.c | 4 +--
6 files changed, 37 insertions(+), 90 deletions(-)
diff --git a/include/linux/compaction.h b/include/linux/compaction.h
index aa8f61c..8d98f3c 100644
--- a/include/linux/compaction.h
+++ b/include/linux/compaction.h
@@ -45,11 +45,8 @@ extern void reset_isolation_suitable(pg_data_t *pgdat);
extern unsigned long compaction_suitable(struct zone *zone, int order,
int alloc_flags, int classzone_idx);
-extern void defer_compaction(struct zone *zone, int order);
-extern bool compaction_deferred(struct zone *zone, int order);
-extern void compaction_defer_reset(struct zone *zone, int order,
+extern void compaction_failed_reset(struct zone *zone, int order,
bool alloc_success);
-extern bool compaction_restarting(struct zone *zone, int order);
#else
static inline unsigned long try_to_compact_pages(gfp_t gfp_mask,
@@ -74,15 +71,6 @@ static inline unsigned long compaction_suitable(struct zone *zone, int order,
return COMPACT_SKIPPED;
}
-static inline void defer_compaction(struct zone *zone, int order)
-{
-}
-
-static inline bool compaction_deferred(struct zone *zone, int order)
-{
- return true;
-}
-
#endif /* CONFIG_COMPACTION */
#if defined(CONFIG_COMPACTION) && defined(CONFIG_SYSFS) && defined(CONFIG_NUMA)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 700e9b5..e13b732 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -514,8 +514,7 @@ struct zone {
* are skipped before trying again. The number attempted since
* last failure is tracked with compact_considered.
*/
- unsigned int compact_considered;
- unsigned int compact_defer_shift;
+ int compact_failed;
int compact_order_failed;
unsigned long compact_success;
unsigned long compact_depletion_depth;
diff --git a/include/trace/events/compaction.h b/include/trace/events/compaction.h
index 9a6a3fe..323e614 100644
--- a/include/trace/events/compaction.h
+++ b/include/trace/events/compaction.h
@@ -239,7 +239,7 @@ DEFINE_EVENT(mm_compaction_suitable_template, mm_compaction_suitable,
);
#ifdef CONFIG_COMPACTION
-DECLARE_EVENT_CLASS(mm_compaction_defer_template,
+DECLARE_EVENT_CLASS(mm_compaction_deplete_template,
TP_PROTO(struct zone *zone, int order),
@@ -249,8 +249,9 @@ DECLARE_EVENT_CLASS(mm_compaction_defer_template,
__field(int, nid)
__field(char *, name)
__field(int, order)
- __field(unsigned int, considered)
- __field(unsigned int, defer_shift)
+ __field(unsigned long, success)
+ __field(unsigned long, depletion_depth)
+ __field(int, failed)
__field(int, order_failed)
),
@@ -258,35 +259,30 @@ DECLARE_EVENT_CLASS(mm_compaction_defer_template,
__entry->nid = zone_to_nid(zone);
__entry->name = (char *)zone->name;
__entry->order = order;
- __entry->considered = zone->compact_considered;
- __entry->defer_shift = zone->compact_defer_shift;
+ __entry->success = zone->compact_success;
+ __entry->depletion_depth = zone->compact_depletion_depth;
+ __entry->failed = zone->compact_failed;
__entry->order_failed = zone->compact_order_failed;
),
- TP_printk("node=%d zone=%-8s order=%d order_failed=%d consider=%u limit=%lu",
+ TP_printk("node=%d zone=%-8s order=%d failed=%d order_failed=%d consider=%lu depth=%lu",
__entry->nid,
__entry->name,
__entry->order,
+ __entry->failed,
__entry->order_failed,
- __entry->considered,
- 1UL << __entry->defer_shift)
+ __entry->success,
+ __entry->depletion_depth)
);
-DEFINE_EVENT(mm_compaction_defer_template, mm_compaction_deferred,
+DEFINE_EVENT(mm_compaction_deplete_template, mm_compaction_fail_compaction,
TP_PROTO(struct zone *zone, int order),
TP_ARGS(zone, order)
);
-DEFINE_EVENT(mm_compaction_defer_template, mm_compaction_defer_compaction,
-
- TP_PROTO(struct zone *zone, int order),
-
- TP_ARGS(zone, order)
-);
-
-DEFINE_EVENT(mm_compaction_defer_template, mm_compaction_defer_reset,
+DEFINE_EVENT(mm_compaction_deplete_template, mm_compaction_failed_reset,
TP_PROTO(struct zone *zone, int order),
diff --git a/mm/compaction.c b/mm/compaction.c
index aff536f..649fca2 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -128,7 +128,7 @@ static struct page *pageblock_pfn_to_page(unsigned long start_pfn,
#ifdef CONFIG_COMPACTION
/* Do not skip compaction more than 64 times */
-#define COMPACT_MAX_DEFER_SHIFT 6
+#define COMPACT_MAX_FAILED 4
#define COMPACT_MIN_DEPLETE_THRESHOLD 1UL
#define COMPACT_MIN_SCAN_LIMIT (pageblock_nr_pages)
@@ -190,61 +190,28 @@ static void set_migration_scan_limit(struct compact_control *cc)
limit >>= zone->compact_depletion_depth;
cc->migration_scan_limit = max(limit, COMPACT_CLUSTER_MAX);
}
-/*
- * Compaction is deferred when compaction fails to result in a page
- * allocation success. 1 << compact_defer_limit compactions are skipped up
- * to a limit of 1 << COMPACT_MAX_DEFER_SHIFT
- */
-void defer_compaction(struct zone *zone, int order)
-{
- zone->compact_considered = 0;
- zone->compact_defer_shift++;
-
- if (order < zone->compact_order_failed)
- zone->compact_order_failed = order;
-
- if (zone->compact_defer_shift > COMPACT_MAX_DEFER_SHIFT)
- zone->compact_defer_shift = COMPACT_MAX_DEFER_SHIFT;
- trace_mm_compaction_defer_compaction(zone, order);
-}
-
-/* Returns true if compaction should be skipped this time */
-bool compaction_deferred(struct zone *zone, int order)
+void fail_compaction(struct zone *zone, int order)
{
- unsigned long defer_limit = 1UL << zone->compact_defer_shift;
-
- if (order < zone->compact_order_failed)
- return false;
-
- /* Avoid possible overflow */
- if (++zone->compact_considered > defer_limit)
- zone->compact_considered = defer_limit;
-
- if (zone->compact_considered >= defer_limit)
- return false;
-
- trace_mm_compaction_deferred(zone, order);
+ if (order < zone->compact_order_failed) {
+ zone->compact_failed = 0;
+ zone->compact_order_failed = order;
+ } else
+ zone->compact_failed++;
- return true;
+ trace_mm_compaction_fail_compaction(zone, order);
}
-/*
- * Update defer tracking counters after successful compaction of given order,
- * which means an allocation either succeeded (alloc_success == true) or is
- * expected to succeed.
- */
-void compaction_defer_reset(struct zone *zone, int order,
+void compaction_failed_reset(struct zone *zone, int order,
bool alloc_success)
{
- if (alloc_success) {
- zone->compact_considered = 0;
- zone->compact_defer_shift = 0;
- }
+ if (alloc_success)
+ zone->compact_failed = 0;
+
if (order >= zone->compact_order_failed)
zone->compact_order_failed = order + 1;
- trace_mm_compaction_defer_reset(zone, order);
+ trace_mm_compaction_failed_reset(zone, order);
}
/* Returns true if restarting compaction after many failures */
@@ -256,8 +223,7 @@ static bool compaction_direct_restarting(struct zone *zone, int order)
if (order < zone->compact_order_failed)
return false;
- return zone->compact_defer_shift == COMPACT_MAX_DEFER_SHIFT &&
- zone->compact_considered >= 1UL << zone->compact_defer_shift;
+ return zone->compact_failed < COMPACT_MAX_FAILED ? false : true;
}
/* Returns true if the pageblock should be scanned for pages to isolate. */
@@ -295,6 +261,7 @@ static void __reset_isolation_suitable(struct zone *zone)
}
}
zone->compact_success = 0;
+ zone->compact_failed = 0;
/* Walk the zone and mark every pageblock as suitable for isolation */
for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) {
@@ -1610,9 +1577,6 @@ unsigned long try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
int status;
int zone_contended;
- if (compaction_deferred(zone, order))
- continue;
-
status = compact_zone_order(zone, order, gfp_mask, mode,
&zone_contended, alloc_flags,
ac->classzone_idx);
@@ -1632,7 +1596,7 @@ unsigned long try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
* will repeat this with true if allocation indeed
* succeeds in this zone.
*/
- compaction_defer_reset(zone, order, false);
+ compaction_failed_reset(zone, order, false);
/*
* It is possible that async compaction aborted due to
* need_resched() and the watermarks were ok thanks to
@@ -1653,7 +1617,7 @@ unsigned long try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
* so we defer compaction there. If it ends up
* succeeding after all, it will be reset.
*/
- defer_compaction(zone, order);
+ fail_compaction(zone, order);
}
/*
@@ -1715,13 +1679,13 @@ static void __compact_pgdat(pg_data_t *pgdat, struct compact_control *cc)
if (cc->order == -1)
__reset_isolation_suitable(zone);
- if (cc->order == -1 || !compaction_deferred(zone, cc->order))
+ if (cc->order == -1)
compact_zone(zone, cc);
if (cc->order > 0) {
if (zone_watermark_ok(zone, cc->order,
low_wmark_pages(zone), 0, 0))
- compaction_defer_reset(zone, cc->order, false);
+ compaction_failed_reset(zone, cc->order, false);
}
VM_BUG_ON(!list_empty(&cc->freepages));
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index afd5459..f53d764 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2821,7 +2821,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
struct zone *zone = page_zone(page);
zone->compact_blockskip_flush = false;
- compaction_defer_reset(zone, order, true);
+ compaction_failed_reset(zone, order, true);
count_vm_event(COMPACTSUCCESS);
return page;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 37e90db..a561b5f 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -2469,10 +2469,10 @@ static inline bool compaction_ready(struct zone *zone, int order)
watermark_ok = zone_watermark_ok_safe(zone, 0, watermark, 0, 0);
/*
- * If compaction is deferred, reclaim up to a point where
+ * If compaction is depleted, reclaim up to a point where
* compaction will have a chance of success when re-enabled
*/
- if (compaction_deferred(zone, order))
+ if (test_bit(ZONE_COMPACTION_DEPLETED, &zone->flags))
return watermark_ok;
/*
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Joonsoo Kim <iamjoonsoo.kim@lge.com> |
|---|---|
| Date | 2015-06-25 02:50 +0200 |
| Subject | [RFC PATCH 03/10] mm/compaction: always update cached pfn |
| Message-ID | <pF3wC-W9-19@gated-at.bofh.it> |
| In reply to | #1171782 |
Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
---
mm/compaction.c | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/mm/compaction.c b/mm/compaction.c
index 9c5d43c..2d8e211 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -510,6 +510,10 @@ isolate_fail:
if (locked)
spin_unlock_irqrestore(&cc->zone->lock, flags);
+ if (blockpfn == end_pfn &&
+ blockpfn > cc->zone->compact_cached_free_pfn)
+ cc->zone->compact_cached_free_pfn = blockpfn;
+
update_pageblock_skip(cc, valid_page, total_isolated,
*start_pfn, end_pfn, blockpfn, false);
@@ -811,6 +815,13 @@ isolate_success:
if (locked)
spin_unlock_irqrestore(&zone->lru_lock, flags);
+ if (low_pfn == end_pfn && cc->mode != MIGRATE_ASYNC) {
+ int sync = cc->mode != MIGRATE_ASYNC;
+
+ if (low_pfn > zone->compact_cached_migrate_pfn[sync])
+ zone->compact_cached_migrate_pfn[sync] = low_pfn;
+ }
+
update_pageblock_skip(cc, valid_page, nr_isolated,
start_pfn, end_pfn, low_pfn, true);
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2015-06-25 11:10 +0200 |
| Subject | Re: [RFC PATCH 03/10] mm/compaction: always update cached pfn |
| Message-ID | <pFbku-4eT-3@gated-at.bofh.it> |
| In reply to | #1171785 |
I can has commit log? :)
On 06/25/2015 02:45 AM, Joonsoo Kim wrote:
> Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
> ---
> mm/compaction.c | 11 +++++++++++
> 1 file changed, 11 insertions(+)
>
> diff --git a/mm/compaction.c b/mm/compaction.c
> index 9c5d43c..2d8e211 100644
> --- a/mm/compaction.c
> +++ b/mm/compaction.c
> @@ -510,6 +510,10 @@ isolate_fail:
> if (locked)
> spin_unlock_irqrestore(&cc->zone->lock, flags);
>
> + if (blockpfn == end_pfn &&
> + blockpfn > cc->zone->compact_cached_free_pfn)
> + cc->zone->compact_cached_free_pfn = blockpfn;
> +
> update_pageblock_skip(cc, valid_page, total_isolated,
> *start_pfn, end_pfn, blockpfn, false);
>
> @@ -811,6 +815,13 @@ isolate_success:
> if (locked)
> spin_unlock_irqrestore(&zone->lru_lock, flags);
>
> + if (low_pfn == end_pfn && cc->mode != MIGRATE_ASYNC) {
> + int sync = cc->mode != MIGRATE_ASYNC;
> +
> + if (low_pfn > zone->compact_cached_migrate_pfn[sync])
> + zone->compact_cached_migrate_pfn[sync] = low_pfn;
> + }
> +
> update_pageblock_skip(cc, valid_page, nr_isolated,
> start_pfn, end_pfn, low_pfn, true);
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Joonsoo Kim <iamjoonsoo.kim@lge.com> |
|---|---|
| Date | 2015-06-25 02:50 +0200 |
| Subject | [RFC PATCH 05/10] mm/compaction: make freepage scanner scans non-movable pageblock |
| Message-ID | <pF3wC-W9-21@gated-at.bofh.it> |
| In reply to | #1171782 |
Currently, freescanner doesn't scan non-movable pageblock, because if
freepages in non-movable pageblock are exhausted, another movable
pageblock would be used for non-movable allocation and it could cause
fragmentation.
But, we should know that watermark check for compaction doesn't consider
this reality. So, if all freepages are in non-movable pageblock, although,
system has enough freepages and watermark check is passed, freepage
scanner can't get any freepage and compaction will be failed. There is
no way to get precise number of freepage on movable pageblock and no way
to reclaim only used pages in movable pageblock. Therefore, I think
that best way to overcome this situation is to use freepage in non-movable
pageblock in compaction.
My test setup for this situation is:
Memory is artificially fragmented to make order 3 allocation hard. And,
most of pageblocks are changed to unmovable migratetype.
System: 512 MB with 32 MB Zram
Memory: 25% memory is allocated to make fragmentation and kernel build
is running on background.
Fragmentation: Successful order 3 allocation candidates may be around
1500 roughly.
Allocation attempts: Roughly 3000 order 3 allocation attempts
with GFP_NORETRY. This value is determined to saturate allocation
success.
Below is the result of this test.
Test: build-frag-unmovable
base nonmovable
compact_free_scanned 5032378 4110920
compact_isolated 53368 330762
compact_migrate_scanned 1456516 6164677
compact_stall 538 746
compact_success 93 350
pgmigrate_success 19926 152754
Success: 15 31
Success(N): 33 65
Column 'Success' and 'Success(N) are calculated by following equations.
Success = successful allocation * 100 / attempts
Success(N) = successful allocation * 100 / order 3 candidate
Result shows that success rate is doubled in this case
because we can search more area.
But, we can observe regression in other case.
Test: stress-highalloc in mmtests
(tweaks to request order-7 unmovable allocation)
Ops 1 30.00 8.33
Ops 2 32.33 26.67
Ops 3 91.67 92.00
Compaction stalls 5110 5581
Compaction success 1787 1807
Compaction failures 3323 3774
Compaction pages isolated 6370911 15421622
Compaction migrate scanned 52681405 83721428
Compaction free scanned 418049611 579768237
Compaction cost 3745 8822
Although this regression is bad, there are also much improvement
in other cases that most of pageblocks are non-movable migratetype.
IMHO, this patch can be justified by this improvement. Moreover,
this regression disappears after applying following patches, so
we don't need to worry about regression much.
Migration scanner already scans non-movable pageblock and make some
freepage in that pageblock through migration. So, even if freepage
scanner scans non-movable pageblock and uses freepage in that pageblock,
number of freepages on non-movable pageblock wouldn't diminish much and
wouldn't cause much fragmentation.
Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
---
mm/compaction.c | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index dd2063b..8d1b3b5 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -905,12 +905,8 @@ static bool suitable_migration_target(struct page *page)
return false;
}
- /* If the block is MIGRATE_MOVABLE or MIGRATE_CMA, allow migration */
- if (migrate_async_suitable(get_pageblock_migratetype(page)))
- return true;
-
- /* Otherwise skip the block */
- return false;
+ /* Otherwise scan the block */
+ return true;
}
/*
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Joonsoo Kim <iamjoonsoo.kim@lge.com> |
|---|---|
| Date | 2015-06-25 02:50 +0200 |
| Subject | [RFC PATCH 01/10] mm/compaction: update skip-bit if whole pageblock is really scanned |
| Message-ID | <pF3wC-W9-23@gated-at.bofh.it> |
| In reply to | #1171782 |
Scanning pageblock is stopped at the middle of pageblock if enough
pages are isolated. In the next run, it begins again at this position
and if it find that there is no isolation candidate from the middle of
pageblock to end of pageblock, it updates skip-bit. In this case,
scanner doesn't start at begin of pageblock so it is not appropriate
to set skipbit. This patch fixes this situation that updating skip-bit
only happens when whole pageblock is really scanned.
Signed-off-by: Joonsoo Kim <iamjoonsoo.kim@lge.com>
---
mm/compaction.c | 32 ++++++++++++++++++--------------
1 file changed, 18 insertions(+), 14 deletions(-)
diff --git a/mm/compaction.c b/mm/compaction.c
index 6ef2fdf..4397bf7 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -261,7 +261,8 @@ void reset_isolation_suitable(pg_data_t *pgdat)
*/
static void update_pageblock_skip(struct compact_control *cc,
struct page *page, unsigned long nr_isolated,
- bool migrate_scanner)
+ unsigned long start_pfn, unsigned long end_pfn,
+ unsigned long curr_pfn, bool migrate_scanner)
{
struct zone *zone = cc->zone;
unsigned long pfn;
@@ -275,6 +276,13 @@ static void update_pageblock_skip(struct compact_control *cc,
if (nr_isolated)
return;
+ /* Update the pageblock-skip if the whole pageblock was scanned */
+ if (curr_pfn != end_pfn)
+ return;
+
+ if (start_pfn != round_down(end_pfn - 1, pageblock_nr_pages))
+ return;
+
set_pageblock_skip(page);
pfn = page_to_pfn(page);
@@ -300,7 +308,8 @@ static inline bool isolation_suitable(struct compact_control *cc,
static void update_pageblock_skip(struct compact_control *cc,
struct page *page, unsigned long nr_isolated,
- bool migrate_scanner)
+ unsigned long start_pfn, unsigned long end_pfn,
+ unsigned long curr_pfn, bool migrate_scanner)
{
}
#endif /* CONFIG_COMPACTION */
@@ -493,9 +502,6 @@ isolate_fail:
trace_mm_compaction_isolate_freepages(*start_pfn, blockpfn,
nr_scanned, total_isolated);
- /* Record how far we have got within the block */
- *start_pfn = blockpfn;
-
/*
* If strict isolation is requested by CMA then check that all the
* pages requested were isolated. If there were any failures, 0 is
@@ -507,9 +513,11 @@ isolate_fail:
if (locked)
spin_unlock_irqrestore(&cc->zone->lock, flags);
- /* Update the pageblock-skip if the whole pageblock was scanned */
- if (blockpfn == end_pfn)
- update_pageblock_skip(cc, valid_page, total_isolated, false);
+ update_pageblock_skip(cc, valid_page, total_isolated,
+ *start_pfn, end_pfn, blockpfn, false);
+
+ /* Record how far we have got within the block */
+ *start_pfn = blockpfn;
count_compact_events(COMPACTFREE_SCANNED, nr_scanned);
if (total_isolated)
@@ -806,12 +814,8 @@ isolate_success:
if (locked)
spin_unlock_irqrestore(&zone->lru_lock, flags);
- /*
- * Update the pageblock-skip information and cached scanner pfn,
- * if the whole pageblock was scanned without isolating any page.
- */
- if (low_pfn == end_pfn)
- update_pageblock_skip(cc, valid_page, nr_isolated, true);
+ update_pageblock_skip(cc, valid_page, nr_isolated,
+ start_pfn, end_pfn, low_pfn, true);
trace_mm_compaction_isolate_migratepages(start_pfn, low_pfn,
nr_scanned, nr_isolated);
--
1.9.1
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Joonsoo Kim <js1304@gmail.com> |
|---|---|
| Date | 2015-06-26 04:10 +0200 |
| Message-ID | <pFrfA-1Hy-11@gated-at.bofh.it> |
| In reply to | #1171782 |
2015-06-26 3:41 GMT+09:00 Mel Gorman <mgorman@suse.de>: > On Fri, Jun 26, 2015 at 03:14:39AM +0900, Joonsoo Kim wrote: >> > It could though. Reclaim/compaction is entered for orders higher than >> > PAGE_ALLOC_COSTLY_ORDER and when scan priority is sufficiently high. >> > That could be adjusted if you have a viable case where orders < >> > PAGE_ALLOC_COSTLY_ORDER must succeed and currently requires excessive >> > reclaim instead of relying on compaction. >> >> Yes. I saw this problem in real situation. In ARM, order-2 allocation >> is requested >> in fork(), so it should be succeed. But, there is not enough order-2 freepage, >> so reclaim/compaction begins. Compaction fails repeatedly although >> I didn't check exact reason. > > That should be identified and repaired prior to reimplementing > compaction because it's important. Unfortunately, I got report a long time ago and I don't have any real environment to reproduce it. What I have remembered is that there are too many unmovable allocations from graphic driver and zram and they really makes fragmented memory. In that time, problem is solved by ad-hoc approach such as killing many apps. But, it's sub-optimal and loosing performance greatly so I imitate this effect in my benchmark and try to solve it by this patchset. >> >> >> 3) Compaction capability is highly depends on migratetype of memory, >> >> >> because freepage scanner doesn't scan unmovable pageblock. >> >> >> >> >> > >> >> > For a very good reason. Unmovable allocation requests that fallback to >> >> > other pageblocks are the worst in terms of fragmentation avoidance. The >> >> > more of these events there are, the more the system will decay. If there >> >> > are many of these events then a compaction benchmark may start with high >> >> > success rates but decay over time. >> >> > >> >> > Very broadly speaking, the more the mm_page_alloc_extfrag tracepoint >> >> > triggers with alloc_migratetype == MIGRATE_UNMOVABLE, the faster the >> >> > system is decaying. Having the freepage scanner select unmovable >> >> > pageblocks will trigger this event more frequently. >> >> > >> >> > The unfortunate impact is that selecting unmovable blocks from the free >> >> > csanner will improve compaction success rates for high-order kernel >> >> > allocations early in the lifetime of the system but later fail high-order >> >> > allocation requests as more pageblocks get converted to unmovable. It >> >> > might be ok for kernel allocations but THP will eventually have a 100% >> >> > failure rate. >> >> >> >> I wrote rationale in the patch itself. We already use non-movable pageblock >> >> for migration scanner. It empties non-movable pageblock so number of >> >> freepage on non-movable pageblock will increase. Using non-movable >> >> pageblock for freepage scanner negates this effect so number of freepage >> >> on non-movable pageblock will be balanced. Could you tell me in detail >> >> how freepage scanner select unmovable pageblocks will cause >> >> more fragmentation? Possibly, I don't understand effect of this patch >> >> correctly and need some investigation. :) >> >> >> > >> > The long-term success rate of fragmentation avoidance depends on >> > minimsing the number of UNMOVABLE allocation requests that use a >> > pageblock belonging to another migratetype. Once such a fallback occurs, >> > that pageblock potentially can never be used for a THP allocation again. >> > >> > Lets say there is an unmovable pageblock with 500 free pages in it. If >> > the freepage scanner uses that pageblock and allocates all 500 free >> > pages then the next unmovable allocation request needs a new pageblock. >> > If one is not completely free then it will fallback to using a >> > RECLAIMABLE or MOVABLE pageblock forever contaminating it. >> >> Yes, I can imagine that situation. But, as I said above, we already use >> non-movable pageblock for migration scanner. While unmovable >> pageblock with 500 free pages fills, some other unmovable pageblock >> with some movable pages will be emptied. Number of freepage >> on non-movable would be maintained so fallback doesn't happen. >> >> Anyway, it is better to investigate this effect. I will do it and attach >> result on next submission. >> > > Lets say we have X unmovable pageblocks and Y pageblocks overall. If the > migration scanner takes movable pages from X then there is more space for > unmovable allocations without having to increase X -- this is good. If > the free scanner uses the X pageblocks as targets then they can fill. The > next unmovable allocation then falls back to another pageblock and we > either have X+1 unmovable pageblocks (full steal) or a mixed pageblock > (partial steal) that cannot be used for THP. Do this enough times and > X == Y and all THP allocations fail. This was similar with my understanding but different conclusion. As number of unmovable pageblocks, X, which is filled by movable pages due to this compaction change increases, reclaimed/migrated out pages from them also increase. And, then, further unmovable allocation request will use this free space and eventually these pageblocks are totally filled by unmovable allocation. Therefore, I guess, in the long-term, increasing X is saturated and X == Y will not happen. Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [next] | [standalone]
| From | Joonsoo Kim <js1304@gmail.com> |
|---|---|
| Date | 2015-06-26 04:20 +0200 |
| Message-ID | <pFrpg-1YQ-17@gated-at.bofh.it> |
| In reply to | #1171782 |
2015-06-26 3:56 GMT+09:00 Vlastimil Babka <vbabka@suse.cz>: > On 25.6.2015 20:14, Joonsoo Kim wrote: >>> The long-term success rate of fragmentation avoidance depends on >>> > minimsing the number of UNMOVABLE allocation requests that use a >>> > pageblock belonging to another migratetype. Once such a fallback occurs, >>> > that pageblock potentially can never be used for a THP allocation again. >>> > >>> > Lets say there is an unmovable pageblock with 500 free pages in it. If >>> > the freepage scanner uses that pageblock and allocates all 500 free >>> > pages then the next unmovable allocation request needs a new pageblock. >>> > If one is not completely free then it will fallback to using a >>> > RECLAIMABLE or MOVABLE pageblock forever contaminating it. >> Yes, I can imagine that situation. But, as I said above, we already use >> non-movable pageblock for migration scanner. While unmovable >> pageblock with 500 free pages fills, some other unmovable pageblock >> with some movable pages will be emptied. Number of freepage >> on non-movable would be maintained so fallback doesn't happen. > > There's nothing that guarantees that the migration scanner will be emptying > unmovable pageblock, or am I missing something? As replied to Mel's comment, as number of unmovable pageblocks, which is filled by movable pages due to this compaction change increases, possible candidate reclaimable/migratable pages from them also increase. So, at some time, amount of used page by free scanner and amount of migrated page by migration scanner would be balanced. > Worse, those pageblocks would be > marked to skip by the free scanner if it isolated free pages from them, so > migration scanner would skip them. Yes, but, next iteration will move out movable pages from that pageblock and freed pages will be used for further unmovable allocation. So, in the long term, this doesn't make much more fragmentation. Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web