Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1379561 > unrolled thread
| Started by | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| First post | 2016-04-15 11:00 +0200 |
| Last post | 2016-04-26 13:50 +0200 |
| Articles | 20 on this page of 70 — 4 participants |
Back to article view | Back to linux.kernel
[PATCH 00/28] Optimise page alloc/free fast paths v3 Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:00 +0200
[PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:00 +0200
Re: [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages Vlastimil Babka <vbabka@suse.cz> - 2016-04-25 11:40 +0200
Re: [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages Mel Gorman <mgorman@techsingularity.net> - 2016-04-26 12:40 +0200
Re: [PATCH 01/28] mm, page_alloc: Only check PageCompound for high-order pages Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:30 +0200
[PATCH 21/28] mm, page_alloc: Avoid looking up the first zone in a zonelist twice Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 21/28] mm, page_alloc: Avoid looking up the first zone in a zonelist twice Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 19:50 +0200
[PATCH 04/28] mm, page_alloc: Inline zone_statistics Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 04/28] mm, page_alloc: Inline zone_statistics Vlastimil Babka <vbabka@suse.cz> - 2016-04-25 13:20 +0200
[PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 15:50 +0200
Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Mel Gorman <mgorman@techsingularity.net> - 2016-04-26 17:00 +0200
Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 17:20 +0200
Re: [PATCH 15/28] mm, page_alloc: Move might_sleep_if check to the allocator slowpath Mel Gorman <mgorman@techsingularity.net> - 2016-04-26 18:30 +0200
[PATCH 19/28] mm, page_alloc: Reduce cost of fair zone allocation policy retry Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 22/28] mm, page_alloc: Remove field from alloc_context Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 14/28] mm, page_alloc: Simplify last cpupid reset Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 14/28] mm, page_alloc: Simplify last cpupid reset Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 15:40 +0200
[PATCH 20/28] mm, page_alloc: Shortcut watermark checks for order-0 pages Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 16/28] mm, page_alloc: Move __GFP_HARDWALL modifications out of the fastpath Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 16/28] mm, page_alloc: Move __GFP_HARDWALL modifications out of the fastpath Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 16:20 +0200
[PATCH 17/28] mm, page_alloc: Check once if a zone has isolated pageblocks Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 17/28] mm, page_alloc: Check once if a zone has isolated pageblocks Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 16:30 +0200
[PATCH 18/28] mm, page_alloc: Shorten the page allocator fast path Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 18/28] mm, page_alloc: Shorten the page allocator fast path Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 17:30 +0200
[PATCH 26/28] cpuset: use static key better and convert to new API Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
Re: [PATCH 26/28] cpuset: use static key better and convert to new API Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 22:00 +0200
[PATCH 27/28] mm, page_alloc: Defer debugging checks of freed pages until a PCP drain Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
Re: [PATCH 27/28] mm, page_alloc: Defer debugging checks of freed pages until a PCP drain Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 14:00 +0200
[PATCH 2/3] mm, page_alloc: pull out side effects from free_pages_check Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 14:10 +0200
Re: [PATCH 2/3] mm, page_alloc: pull out side effects from free_pages_check Mel Gorman <mgorman@techsingularity.net> - 2016-04-27 14:50 +0200
Re: [PATCH 2/3] mm, page_alloc: pull out side effects from free_pages_check Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 15:10 +0200
[PATCH 3/3] mm, page_alloc: don't duplicate code in free_pcp_prepare Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 14:10 +0200
[PATCH 1/3] mm, page_alloc: un-inline the bad part of free_pages_check Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 14:10 +0200
Re: [PATCH 1/3] mm, page_alloc: un-inline the bad part of free_pages_check Mel Gorman <mgorman@techsingularity.net> - 2016-04-27 14:40 +0200
Re: [PATCH 1/3] mm, page_alloc: un-inline the bad part of free_pages_check Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 15:00 +0200
[PATCH 25/28] mm, page_alloc: Inline pageblock lookup in page free fast paths Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
[PATCH 24/28] mm, page_alloc: Remove unnecessary variable from free_pcppages_bulk Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
[PATCH 23/28] mm, page_alloc: Check multiple page fields with a single branch Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
Re: [PATCH 23/28] mm, page_alloc: Check multiple page fields with a single branch Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 20:50 +0200
Re: [PATCH 23/28] mm, page_alloc: Check multiple page fields with a single branch Mel Gorman <mgorman@techsingularity.net> - 2016-04-27 12:10 +0200
[PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:20 +0200
Re: [PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP Vlastimil Babka <vbabka@suse.cz> - 2016-04-27 16:10 +0200
Re: [PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP Mel Gorman <mgorman@techsingularity.net> - 2016-04-27 17:40 +0200
Re: [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 14:10 +0200
Re: [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-26 15:10 +0200
Re: [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist Andrew Morton <akpm@linux-foundation.org> - 2016-04-26 21:20 +0200
[PATCH 07/28] mm, page_alloc: Avoid unnecessary zone lookups during pageblock operations Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 07/28] mm, page_alloc: Avoid unnecessary zone lookups during pageblock operations Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:30 +0200
[PATCH 02/28] mm, page_alloc: Use new PageAnonHead helper in the free page fast path Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 02/28] mm, page_alloc: Use new PageAnonHead helper in the free page fast path Vlastimil Babka <vbabka@suse.cz> - 2016-04-25 12:00 +0200
[PATCH 11/28] mm, page_alloc: Remove unnecessary initialisation in get_page_from_freelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 06/28] mm, page_alloc: Use __dec_zone_state for order-0 page allocation Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 06/28] mm, page_alloc: Use __dec_zone_state for order-0 page allocation Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:30 +0200
[PATCH 09/28] mm, page_alloc: Convert nr_fair_skipped to bool Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 09/28] mm, page_alloc: Convert nr_fair_skipped to bool Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:40 +0200
[PATCH 08/28] mm, page_alloc: Convert alloc_flags to unsigned Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
[PATCH 03/28] mm, page_alloc: Reduce branches in zone_statistics Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 03/28] mm, page_alloc: Reduce branches in zone_statistics Vlastimil Babka <vbabka@suse.cz> - 2016-04-25 13:20 +0200
[PATCH 05/28] mm, page_alloc: Inline the fast path of the zonelist iterator Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 05/28] mm, page_alloc: Inline the fast path of the zonelist iterator Vlastimil Babka <vbabka@suse.cz> - 2016-04-25 17:00 +0200
Re: [PATCH 05/28] mm, page_alloc: Inline the fast path of the zonelist iterator Mel Gorman <mgorman@techsingularity.net> - 2016-04-26 12:40 +0200
Re: [PATCH 05/28] mm, page_alloc: Inline the fast path of the zonelist iterator Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:10 +0200
[PATCH 10/28] mm, page_alloc: Remove unnecessary local variable in get_page_from_freelist Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 11:10 +0200
Re: [PATCH 10/28] mm, page_alloc: Remove unnecessary local variable in get_page_from_freelist Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:40 +0200
Re: [PATCH 00/28] Optimise page alloc/free fast paths v3 Jesper Dangaard Brouer <brouer@redhat.com> - 2016-04-15 14:50 +0200
Re: [PATCH 00/28] Optimise page alloc/free fast paths v3 Mel Gorman <mgorman@techsingularity.net> - 2016-04-15 15:10 +0200
[PATCH 12/28] mm, page_alloc: Remove unnecessary initialisation from __alloc_pages_nodemask() Mel Gorman <mgorman@techsingularity.net> - 2016-04-16 09:30 +0200
Re: [PATCH 12/28] mm, page_alloc: Remove unnecessary initialisation from __alloc_pages_nodemask() Vlastimil Babka <vbabka@suse.cz> - 2016-04-26 13:50 +0200
Page 3 of 4 — ← Prev page 1 2 [3] 4 Next page →
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-26 20:50 +0200 |
| Subject | Re: [PATCH 23/28] mm, page_alloc: Check multiple page fields with a single branch |
| Message-ID | <rsgdB-42k-47@gated-at.bofh.it> |
| In reply to | #1379614 |
On 04/15/2016 11:07 AM, Mel Gorman wrote: > Every page allocated or freed is checked for sanity to avoid corruptions > that are difficult to detect later. A bad page could be due to a number of > fields. Instead of using multiple branches, this patch combines multiple > fields into a single branch. A detailed check is only necessary if that > check fails. > > 4.6.0-rc2 4.6.0-rc2 > initonce-v1r20 multcheck-v1r20 > Min alloc-odr0-1 359.00 ( 0.00%) 348.00 ( 3.06%) > Min alloc-odr0-2 260.00 ( 0.00%) 254.00 ( 2.31%) > Min alloc-odr0-4 214.00 ( 0.00%) 213.00 ( 0.47%) > Min alloc-odr0-8 186.00 ( 0.00%) 186.00 ( 0.00%) > Min alloc-odr0-16 173.00 ( 0.00%) 173.00 ( 0.00%) > Min alloc-odr0-32 165.00 ( 0.00%) 166.00 ( -0.61%) > Min alloc-odr0-64 162.00 ( 0.00%) 162.00 ( 0.00%) > Min alloc-odr0-128 161.00 ( 0.00%) 160.00 ( 0.62%) > Min alloc-odr0-256 170.00 ( 0.00%) 169.00 ( 0.59%) > Min alloc-odr0-512 181.00 ( 0.00%) 180.00 ( 0.55%) > Min alloc-odr0-1024 190.00 ( 0.00%) 188.00 ( 1.05%) > Min alloc-odr0-2048 196.00 ( 0.00%) 194.00 ( 1.02%) > Min alloc-odr0-4096 202.00 ( 0.00%) 199.00 ( 1.49%) > Min alloc-odr0-8192 205.00 ( 0.00%) 202.00 ( 1.46%) > Min alloc-odr0-16384 205.00 ( 0.00%) 203.00 ( 0.98%) > > Again, the benefit is marginal but avoiding excessive branches is > important. Ideally the paths would not have to check these conditions at > all but regrettably abandoning the tests would make use-after-free bugs > much harder to detect. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Vlastimil Babka <vbabka@suse.cz> I wonder, would it be just too ugly to add +1 to atomic_read(&page->_mapcount) and OR it with the rest for a truly single branch?
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-27 12:10 +0200 |
| Subject | Re: [PATCH 23/28] mm, page_alloc: Check multiple page fields with a single branch |
| Message-ID | <rsuzW-7RS-45@gated-at.bofh.it> |
| In reply to | #1387891 |
On Tue, Apr 26, 2016 at 08:41:50PM +0200, Vlastimil Babka wrote: > On 04/15/2016 11:07 AM, Mel Gorman wrote: > >Signed-off-by: Mel Gorman <mgorman@techsingularity.net> > > Acked-by: Vlastimil Babka <vbabka@suse.cz> > > I wonder, would it be just too ugly to add +1 to > atomic_read(&page->_mapcount) and OR it with the rest for a truly single > branch? > Interesting thought. I'm not going to do it as a fix but when I'm doing the next round of page allocator material, I'll add it to the pile for evaluation. -- Mel Gorman SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:20 +0200 |
| Subject | [PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP |
| Message-ID | <ro850-In-85@gated-at.bofh.it> |
| In reply to | #1379570 |
Every page allocated checks a number of page fields for validity. This
catches corruption bugs of pages that are already freed but it is expensive.
This patch weakens the debugging check by checking PCP pages only when
the PCP lists are being refilled. All compound pages are checked. This
potentially avoids debugging checks entirely if the PCP lists are never
emptied and refilled so some corruption issues may be missed. Full checking
requires DEBUG_VM.
With the two deferred debugging patches applied, the impact to a page
allocator microbenchmark is
4.6.0-rc3 4.6.0-rc3
inline-v3r6 deferalloc-v3r7
Min alloc-odr0-1 344.00 ( 0.00%) 317.00 ( 7.85%)
Min alloc-odr0-2 248.00 ( 0.00%) 231.00 ( 6.85%)
Min alloc-odr0-4 209.00 ( 0.00%) 192.00 ( 8.13%)
Min alloc-odr0-8 181.00 ( 0.00%) 166.00 ( 8.29%)
Min alloc-odr0-16 168.00 ( 0.00%) 154.00 ( 8.33%)
Min alloc-odr0-32 161.00 ( 0.00%) 148.00 ( 8.07%)
Min alloc-odr0-64 158.00 ( 0.00%) 145.00 ( 8.23%)
Min alloc-odr0-128 156.00 ( 0.00%) 143.00 ( 8.33%)
Min alloc-odr0-256 168.00 ( 0.00%) 154.00 ( 8.33%)
Min alloc-odr0-512 178.00 ( 0.00%) 167.00 ( 6.18%)
Min alloc-odr0-1024 186.00 ( 0.00%) 174.00 ( 6.45%)
Min alloc-odr0-2048 192.00 ( 0.00%) 180.00 ( 6.25%)
Min alloc-odr0-4096 198.00 ( 0.00%) 184.00 ( 7.07%)
Min alloc-odr0-8192 200.00 ( 0.00%) 188.00 ( 6.00%)
Min alloc-odr0-16384 201.00 ( 0.00%) 188.00 ( 6.47%)
Min free-odr0-1 189.00 ( 0.00%) 180.00 ( 4.76%)
Min free-odr0-2 132.00 ( 0.00%) 126.00 ( 4.55%)
Min free-odr0-4 104.00 ( 0.00%) 99.00 ( 4.81%)
Min free-odr0-8 90.00 ( 0.00%) 85.00 ( 5.56%)
Min free-odr0-16 84.00 ( 0.00%) 80.00 ( 4.76%)
Min free-odr0-32 80.00 ( 0.00%) 76.00 ( 5.00%)
Min free-odr0-64 78.00 ( 0.00%) 74.00 ( 5.13%)
Min free-odr0-128 77.00 ( 0.00%) 73.00 ( 5.19%)
Min free-odr0-256 94.00 ( 0.00%) 91.00 ( 3.19%)
Min free-odr0-512 108.00 ( 0.00%) 112.00 ( -3.70%)
Min free-odr0-1024 115.00 ( 0.00%) 118.00 ( -2.61%)
Min free-odr0-2048 120.00 ( 0.00%) 125.00 ( -4.17%)
Min free-odr0-4096 123.00 ( 0.00%) 129.00 ( -4.88%)
Min free-odr0-8192 126.00 ( 0.00%) 130.00 ( -3.17%)
Min free-odr0-16384 126.00 ( 0.00%) 131.00 ( -3.97%)
Note that the free paths for large numbers of pages is impacted as the
debugging cost gets shifted into that path when the page data is no longer
necessarily cache-hot.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/page_alloc.c | 92 +++++++++++++++++++++++++++++++++++++++------------------
1 file changed, 64 insertions(+), 28 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index b5722790c846..147c0d55ed32 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1704,7 +1704,41 @@ static inline bool free_pages_prezeroed(bool poisoned)
page_poisoning_enabled() && poisoned;
}
-static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
+#ifdef CONFIG_DEBUG_VM
+static bool check_pcp_refill(struct page *page)
+{
+ return false;
+}
+
+static bool check_new_pcp(struct page *page)
+{
+ return check_new_page(page);
+}
+#else
+static bool check_pcp_refill(struct page *page)
+{
+ return check_new_page(page);
+}
+static bool check_new_pcp(struct page *page)
+{
+ return false;
+}
+#endif /* CONFIG_DEBUG_VM */
+
+static bool check_new_pages(struct page *page, unsigned int order)
+{
+ int i;
+ for (i = 0; i < (1 << order); i++) {
+ struct page *p = page + i;
+
+ if (unlikely(check_new_page(p)))
+ return true;
+ }
+
+ return false;
+}
+
+static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
unsigned int alloc_flags)
{
int i;
@@ -1712,8 +1746,6 @@ static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
for (i = 0; i < (1 << order); i++) {
struct page *p = page + i;
- if (unlikely(check_new_page(p)))
- return 1;
if (poisoned)
poisoned &= page_is_poisoned(p);
}
@@ -1745,8 +1777,6 @@ static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
set_page_pfmemalloc(page);
else
clear_page_pfmemalloc(page);
-
- return 0;
}
/*
@@ -2168,6 +2198,9 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,
if (unlikely(page == NULL))
break;
+ if (unlikely(check_pcp_refill(page)))
+ continue;
+
/*
* Split buddy pages returned by expand() are received here
* in physical page order. The page is added to the callers and
@@ -2579,20 +2612,22 @@ struct page *buffered_rmqueue(struct zone *preferred_zone,
struct list_head *list;
local_irq_save(flags);
- pcp = &this_cpu_ptr(zone->pageset)->pcp;
- list = &pcp->lists[migratetype];
- if (list_empty(list)) {
- pcp->count += rmqueue_bulk(zone, 0,
- pcp->batch, list,
- migratetype, cold);
- if (unlikely(list_empty(list)))
- goto failed;
- }
+ do {
+ pcp = &this_cpu_ptr(zone->pageset)->pcp;
+ list = &pcp->lists[migratetype];
+ if (list_empty(list)) {
+ pcp->count += rmqueue_bulk(zone, 0,
+ pcp->batch, list,
+ migratetype, cold);
+ if (unlikely(list_empty(list)))
+ goto failed;
+ }
- if (cold)
- page = list_last_entry(list, struct page, lru);
- else
- page = list_first_entry(list, struct page, lru);
+ if (cold)
+ page = list_last_entry(list, struct page, lru);
+ else
+ page = list_first_entry(list, struct page, lru);
+ } while (page && check_new_pcp(page));
__dec_zone_state(zone, NR_ALLOC_BATCH);
list_del(&page->lru);
@@ -2605,14 +2640,16 @@ struct page *buffered_rmqueue(struct zone *preferred_zone,
WARN_ON_ONCE((gfp_flags & __GFP_NOFAIL) && (order > 1));
spin_lock_irqsave(&zone->lock, flags);
- page = NULL;
- if (alloc_flags & ALLOC_HARDER) {
- page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
- if (page)
- trace_mm_page_alloc_zone_locked(page, order, migratetype);
- }
- if (!page)
- page = __rmqueue(zone, order, migratetype);
+ do {
+ page = NULL;
+ if (alloc_flags & ALLOC_HARDER) {
+ page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC);
+ if (page)
+ trace_mm_page_alloc_zone_locked(page, order, migratetype);
+ }
+ if (!page)
+ page = __rmqueue(zone, order, migratetype);
+ } while (page && check_new_pages(page, order));
spin_unlock(&zone->lock);
if (!page)
goto failed;
@@ -2979,8 +3016,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
page = buffered_rmqueue(ac->preferred_zoneref->zone, zone, order,
gfp_mask, alloc_flags, ac->migratetype);
if (page) {
- if (prep_new_page(page, order, gfp_mask, alloc_flags))
- goto try_this_zone;
+ prep_new_page(page, order, gfp_mask, alloc_flags);
/*
* If this is a high-order atomic allocation then check
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-27 16:10 +0200 |
| Subject | Re: [PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP |
| Message-ID | <rsyka-2uQ-7@gated-at.bofh.it> |
| In reply to | #1379621 |
On 04/15/2016 11:07 AM, Mel Gorman wrote:
> Every page allocated checks a number of page fields for validity. This
> catches corruption bugs of pages that are already freed but it is expensive.
> This patch weakens the debugging check by checking PCP pages only when
> the PCP lists are being refilled. All compound pages are checked. This
> potentially avoids debugging checks entirely if the PCP lists are never
> emptied and refilled so some corruption issues may be missed. Full checking
> requires DEBUG_VM.
>
> With the two deferred debugging patches applied, the impact to a page
> allocator microbenchmark is
>
> 4.6.0-rc3 4.6.0-rc3
> inline-v3r6 deferalloc-v3r7
> Min alloc-odr0-1 344.00 ( 0.00%) 317.00 ( 7.85%)
> Min alloc-odr0-2 248.00 ( 0.00%) 231.00 ( 6.85%)
> Min alloc-odr0-4 209.00 ( 0.00%) 192.00 ( 8.13%)
> Min alloc-odr0-8 181.00 ( 0.00%) 166.00 ( 8.29%)
> Min alloc-odr0-16 168.00 ( 0.00%) 154.00 ( 8.33%)
> Min alloc-odr0-32 161.00 ( 0.00%) 148.00 ( 8.07%)
> Min alloc-odr0-64 158.00 ( 0.00%) 145.00 ( 8.23%)
> Min alloc-odr0-128 156.00 ( 0.00%) 143.00 ( 8.33%)
> Min alloc-odr0-256 168.00 ( 0.00%) 154.00 ( 8.33%)
> Min alloc-odr0-512 178.00 ( 0.00%) 167.00 ( 6.18%)
> Min alloc-odr0-1024 186.00 ( 0.00%) 174.00 ( 6.45%)
> Min alloc-odr0-2048 192.00 ( 0.00%) 180.00 ( 6.25%)
> Min alloc-odr0-4096 198.00 ( 0.00%) 184.00 ( 7.07%)
> Min alloc-odr0-8192 200.00 ( 0.00%) 188.00 ( 6.00%)
> Min alloc-odr0-16384 201.00 ( 0.00%) 188.00 ( 6.47%)
> Min free-odr0-1 189.00 ( 0.00%) 180.00 ( 4.76%)
> Min free-odr0-2 132.00 ( 0.00%) 126.00 ( 4.55%)
> Min free-odr0-4 104.00 ( 0.00%) 99.00 ( 4.81%)
> Min free-odr0-8 90.00 ( 0.00%) 85.00 ( 5.56%)
> Min free-odr0-16 84.00 ( 0.00%) 80.00 ( 4.76%)
> Min free-odr0-32 80.00 ( 0.00%) 76.00 ( 5.00%)
> Min free-odr0-64 78.00 ( 0.00%) 74.00 ( 5.13%)
> Min free-odr0-128 77.00 ( 0.00%) 73.00 ( 5.19%)
> Min free-odr0-256 94.00 ( 0.00%) 91.00 ( 3.19%)
> Min free-odr0-512 108.00 ( 0.00%) 112.00 ( -3.70%)
> Min free-odr0-1024 115.00 ( 0.00%) 118.00 ( -2.61%)
> Min free-odr0-2048 120.00 ( 0.00%) 125.00 ( -4.17%)
> Min free-odr0-4096 123.00 ( 0.00%) 129.00 ( -4.88%)
> Min free-odr0-8192 126.00 ( 0.00%) 130.00 ( -3.17%)
> Min free-odr0-16384 126.00 ( 0.00%) 131.00 ( -3.97%)
>
> Note that the free paths for large numbers of pages is impacted as the
> debugging cost gets shifted into that path when the page data is no longer
> necessarily cache-hot.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
Unlike the free path, there are no duplications here, which is nice.
Some un-inlining of bad page check should still work here though imho:
From afdefd87f2d8d07cba4bd2a2f3531dc8bb0b7a19 Mon Sep 17 00:00:00 2001
From: Vlastimil Babka <vbabka@suse.cz>
Date: Wed, 27 Apr 2016 15:47:29 +0200
Subject: [PATCH] mm, page_alloc: uninline the bad page part of
check_new_page()
Bad pages should be rare so the code handling them doesn't need to be inline
for performance reasons. Put it to separate function which returns void.
This also assumes that the initial page_expected_state() result will match the
result of the thorough check, i.e. the page doesn't become "good" in the
meanwhile. This matches the same expectations already in place in
free_pages_check().
!DEBUG_VM bloat-o-meter:
add/remove: 1/0 grow/shrink: 0/1 up/down: 134/-274 (-140)
function old new delta
check_new_page_bad - 134 +134
get_page_from_freelist 3468 3194 -274
Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
mm/page_alloc.c | 33 +++++++++++++++++----------------
1 file changed, 17 insertions(+), 16 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 2b3aefdfcaa2..755ec9465d8a 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1648,19 +1648,11 @@ static inline void expand(struct zone *zone, struct page *page,
}
}
-/*
- * This page is about to be returned from the page allocator
- */
-static inline int check_new_page(struct page *page)
+static void check_new_page_bad(struct page *page)
{
- const char *bad_reason;
- unsigned long bad_flags;
+ const char *bad_reason = NULL;
+ unsigned long bad_flags = 0;
- if (page_expected_state(page, PAGE_FLAGS_CHECK_AT_PREP|__PG_HWPOISON))
- return 0;
-
- bad_reason = NULL;
- bad_flags = 0;
if (unlikely(atomic_read(&page->_mapcount) != -1))
bad_reason = "nonzero mapcount";
if (unlikely(page->mapping != NULL))
@@ -1679,11 +1671,20 @@ static inline int check_new_page(struct page *page)
if (unlikely(page->mem_cgroup))
bad_reason = "page still charged to cgroup";
#endif
- if (unlikely(bad_reason)) {
- bad_page(page, bad_reason, bad_flags);
- return 1;
- }
- return 0;
+ bad_page(page, bad_reason, bad_flags);
+}
+
+/*
+ * This page is about to be returned from the page allocator
+ */
+static inline int check_new_page(struct page *page)
+{
+ if (likely(page_expected_state(page,
+ PAGE_FLAGS_CHECK_AT_PREP|__PG_HWPOISON)))
+ return 0;
+
+ check_new_page_bad(page);
+ return 1;
}
static inline bool free_pages_prezeroed(bool poisoned)
--
2.8.1
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-27 17:40 +0200 |
| Subject | Re: [PATCH 28/28] mm, page_alloc: Defer debugging checks of pages allocated from the PCP |
| Message-ID | <rszJj-3tA-85@gated-at.bofh.it> |
| In reply to | #1389093 |
On Wed, Apr 27, 2016 at 04:06:11PM +0200, Vlastimil Babka wrote: > From afdefd87f2d8d07cba4bd2a2f3531dc8bb0b7a19 Mon Sep 17 00:00:00 2001 > From: Vlastimil Babka <vbabka@suse.cz> > Date: Wed, 27 Apr 2016 15:47:29 +0200 > Subject: [PATCH] mm, page_alloc: uninline the bad page part of > check_new_page() > > Bad pages should be rare so the code handling them doesn't need to be inline > for performance reasons. Put it to separate function which returns void. > This also assumes that the initial page_expected_state() result will match the > result of the thorough check, i.e. the page doesn't become "good" in the > meanwhile. This matches the same expectations already in place in > free_pages_check(). > > !DEBUG_VM bloat-o-meter: > > add/remove: 1/0 grow/shrink: 0/1 up/down: 134/-274 (-140) > function old new delta > check_new_page_bad - 134 +134 > get_page_from_freelist 3468 3194 -274 > > Signed-off-by: Vlastimil Babka <vbabka@suse.cz> Acked-by: Mel Gorman <mgorman@techsingularity.net> Andrew, if you pick up v2 of of the follow-up series then can you also add this patch on top if it's convenient please? -- Mel Gorman SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-26 14:10 +0200 |
| Subject | Re: [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist |
| Message-ID | <rs9Yu-7Ev-7@gated-at.bofh.it> |
| In reply to | #1379570 |
On 04/15/2016 11:07 AM, Mel Gorman wrote: > A check is made for an empty zonelist early in the page allocator fast path > but it's unnecessary. When get_page_from_freelist() is called, it'll return > NULL immediately. Removing the first check is slower for machines with > memoryless nodes but that is a corner case that can live with the overhead. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> > --- > mm/page_alloc.c | 11 ----------- > 1 file changed, 11 deletions(-) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index df03ccc7f07c..21aaef6ddd7a 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -3374,14 +3374,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, > if (should_fail_alloc_page(gfp_mask, order)) > return NULL; > > - /* > - * Check the zones suitable for the gfp_mask contain at least one > - * valid zone. It's possible to have an empty zonelist as a result > - * of __GFP_THISNODE and a memoryless node > - */ > - if (unlikely(!zonelist->_zonerefs->zone)) > - return NULL; > - > if (IS_ENABLED(CONFIG_CMA) && ac.migratetype == MIGRATE_MOVABLE) > alloc_flags |= ALLOC_CMA; > > @@ -3394,8 +3386,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, > /* The preferred zone is used for statistics later */ > preferred_zoneref = first_zones_zonelist(ac.zonelist, ac.high_zoneidx, > ac.nodemask, &ac.preferred_zone); > - if (!ac.preferred_zone) > - goto out; Is this part really safe? Besides changelog doesn't mention preferred_zone. What if somebody attempts e.g. a DMA allocation with ac.nodemask being set to cpuset_current_mems_allowed and initially only containing nodes without ZONE_DMA. Then ac.preferred_zone is NULL, yet we proceed to get_page_from_freelist(). Meanwhile cpuset_current_mems_allowed gets changed so in fact it does contains a suitable node, so we manage to get inside for_each_zone_zonelist_nodemask(). Then there's zone_local(ac->preferred_zone, zone), which will defererence the NULL ac->preferred_zone? > ac.classzone_idx = zonelist_zone_idx(preferred_zoneref); > > /* First allocation attempt */ > @@ -3418,7 +3408,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, > > trace_mm_page_alloc(page, order, alloc_mask, ac.migratetype); > > -out: > /* > * When updating a task's mems_allowed, it is possible to race with > * parallel threads in such a way that an allocation can fail while >
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-26 15:10 +0200 |
| Subject | Re: [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist |
| Message-ID | <rsaUA-8ny-61@gated-at.bofh.it> |
| In reply to | #1387410 |
On Tue, Apr 26, 2016 at 02:04:51PM +0200, Vlastimil Babka wrote: > On 04/15/2016 11:07 AM, Mel Gorman wrote: > >A check is made for an empty zonelist early in the page allocator fast path > >but it's unnecessary. When get_page_from_freelist() is called, it'll return > >NULL immediately. Removing the first check is slower for machines with > >memoryless nodes but that is a corner case that can live with the overhead. > > > >Signed-off-by: Mel Gorman <mgorman@techsingularity.net> > >--- > > mm/page_alloc.c | 11 ----------- > > 1 file changed, 11 deletions(-) > > > >diff --git a/mm/page_alloc.c b/mm/page_alloc.c > >index df03ccc7f07c..21aaef6ddd7a 100644 > >--- a/mm/page_alloc.c > >+++ b/mm/page_alloc.c > >@@ -3374,14 +3374,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, > > if (should_fail_alloc_page(gfp_mask, order)) > > return NULL; > > > >- /* > >- * Check the zones suitable for the gfp_mask contain at least one > >- * valid zone. It's possible to have an empty zonelist as a result > >- * of __GFP_THISNODE and a memoryless node > >- */ > >- if (unlikely(!zonelist->_zonerefs->zone)) > >- return NULL; > >- > > if (IS_ENABLED(CONFIG_CMA) && ac.migratetype == MIGRATE_MOVABLE) > > alloc_flags |= ALLOC_CMA; > > > >@@ -3394,8 +3386,6 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, > > /* The preferred zone is used for statistics later */ > > preferred_zoneref = first_zones_zonelist(ac.zonelist, ac.high_zoneidx, > > ac.nodemask, &ac.preferred_zone); > >- if (!ac.preferred_zone) > >- goto out; > > Is this part really safe? Besides changelog doesn't mention preferred_zone. > What if somebody attempts e.g. a DMA allocation with ac.nodemask being set > to cpuset_current_mems_allowed and initially only containing nodes without > ZONE_DMA. Then ac.preferred_zone is NULL, yet we proceed to > get_page_from_freelist(). Meanwhile cpuset_current_mems_allowed gets changed > so in fact it does contains a suitable node, so we manage to get inside > for_each_zone_zonelist_nodemask(). Then there's > zone_local(ac->preferred_zone, zone), which will defererence the NULL > ac->preferred_zone? > You're right, this is a potential problem. I thought of a few solutions but they're not necessarily cheaper than the current code. If Andrew is watching, please drop this patch if possible. Otherwise, I'll post a revert within the next 2 days and find an alternative solution that still saves cycles. -- Mel Gorman SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Andrew Morton <akpm@linux-foundation.org> |
|---|---|
| Date | 2016-04-26 21:20 +0200 |
| Subject | Re: [PATCH 13/28] mm, page_alloc: Remove redundant check for empty zonelist |
| Message-ID | <rsgGB-4BZ-3@gated-at.bofh.it> |
| In reply to | #1387466 |
On Tue, 26 Apr 2016 14:00:11 +0100 Mel Gorman <mgorman@techsingularity.net> wrote: > If Andrew is watching, please drop this patch if possible. Thud.
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 07/28] mm, page_alloc: Avoid unnecessary zone lookups during pageblock operations |
| Message-ID | <ro7Vh-Er-27@gated-at.bofh.it> |
| In reply to | #1379561 |
Pageblocks have an associated bitmap to store migrate types and whether
the pageblock should be skipped during compaction. The bitmap may be
associated with a memory section or a zone but the zone is looked up
unconditionally. The compiler should optimise this away automatically so
this is a cosmetic patch only in many cases.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/page_alloc.c | 22 +++++++++-------------
1 file changed, 9 insertions(+), 13 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index ab16560b76e6..d00847bb1612 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -6759,23 +6759,23 @@ void *__init alloc_large_system_hash(const char *tablename,
}
/* Return a pointer to the bitmap storing bits affecting a block of pages */
-static inline unsigned long *get_pageblock_bitmap(struct zone *zone,
+static inline unsigned long *get_pageblock_bitmap(struct page *page,
unsigned long pfn)
{
#ifdef CONFIG_SPARSEMEM
return __pfn_to_section(pfn)->pageblock_flags;
#else
- return zone->pageblock_flags;
+ return page_zone(page)->pageblock_flags;
#endif /* CONFIG_SPARSEMEM */
}
-static inline int pfn_to_bitidx(struct zone *zone, unsigned long pfn)
+static inline int pfn_to_bitidx(struct page *page, unsigned long pfn)
{
#ifdef CONFIG_SPARSEMEM
pfn &= (PAGES_PER_SECTION-1);
return (pfn >> pageblock_order) * NR_PAGEBLOCK_BITS;
#else
- pfn = pfn - round_down(zone->zone_start_pfn, pageblock_nr_pages);
+ pfn = pfn - round_down(page_zone(page)->zone_start_pfn, pageblock_nr_pages);
return (pfn >> pageblock_order) * NR_PAGEBLOCK_BITS;
#endif /* CONFIG_SPARSEMEM */
}
@@ -6793,14 +6793,12 @@ unsigned long get_pfnblock_flags_mask(struct page *page, unsigned long pfn,
unsigned long end_bitidx,
unsigned long mask)
{
- struct zone *zone;
unsigned long *bitmap;
unsigned long bitidx, word_bitidx;
unsigned long word;
- zone = page_zone(page);
- bitmap = get_pageblock_bitmap(zone, pfn);
- bitidx = pfn_to_bitidx(zone, pfn);
+ bitmap = get_pageblock_bitmap(page, pfn);
+ bitidx = pfn_to_bitidx(page, pfn);
word_bitidx = bitidx / BITS_PER_LONG;
bitidx &= (BITS_PER_LONG-1);
@@ -6822,20 +6820,18 @@ void set_pfnblock_flags_mask(struct page *page, unsigned long flags,
unsigned long end_bitidx,
unsigned long mask)
{
- struct zone *zone;
unsigned long *bitmap;
unsigned long bitidx, word_bitidx;
unsigned long old_word, word;
BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 4);
- zone = page_zone(page);
- bitmap = get_pageblock_bitmap(zone, pfn);
- bitidx = pfn_to_bitidx(zone, pfn);
+ bitmap = get_pageblock_bitmap(page, pfn);
+ bitidx = pfn_to_bitidx(page, pfn);
word_bitidx = bitidx / BITS_PER_LONG;
bitidx &= (BITS_PER_LONG-1);
- VM_BUG_ON_PAGE(!zone_spans_pfn(zone, pfn), page);
+ VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page);
bitidx += end_bitidx;
mask <<= (BITS_PER_LONG - bitidx - 1);
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-26 13:30 +0200 |
| Subject | Re: [PATCH 07/28] mm, page_alloc: Avoid unnecessary zone lookups during pageblock operations |
| Message-ID | <rs9lM-6K3-17@gated-at.bofh.it> |
| In reply to | #1379573 |
On 04/15/2016 10:58 AM, Mel Gorman wrote: > Pageblocks have an associated bitmap to store migrate types and whether > the pageblock should be skipped during compaction. The bitmap may be > associated with a memory section or a zone but the zone is looked up > unconditionally. The compiler should optimise this away automatically so > this is a cosmetic patch only in many cases. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Vlastimil Babka <vbabka@suse.cz>
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 02/28] mm, page_alloc: Use new PageAnonHead helper in the free page fast path |
| Message-ID | <ro7Vh-Er-29@gated-at.bofh.it> |
| In reply to | #1379561 |
The PageAnon check always checks for compound_head but this is a relatively
expensive check if the caller already knows the page is a head page. This
patch creates a helper and uses it in the page free path which only operates
on head pages.
With this patch and "Only check PageCompound for high-order pages", the
performance difference on a page allocator microbenchmark is;
4.6.0-rc2 4.6.0-rc2
vanilla nocompound-v1r20
Min alloc-odr0-1 425.00 ( 0.00%) 417.00 ( 1.88%)
Min alloc-odr0-2 313.00 ( 0.00%) 308.00 ( 1.60%)
Min alloc-odr0-4 257.00 ( 0.00%) 253.00 ( 1.56%)
Min alloc-odr0-8 224.00 ( 0.00%) 221.00 ( 1.34%)
Min alloc-odr0-16 208.00 ( 0.00%) 205.00 ( 1.44%)
Min alloc-odr0-32 199.00 ( 0.00%) 199.00 ( 0.00%)
Min alloc-odr0-64 195.00 ( 0.00%) 193.00 ( 1.03%)
Min alloc-odr0-128 192.00 ( 0.00%) 191.00 ( 0.52%)
Min alloc-odr0-256 204.00 ( 0.00%) 200.00 ( 1.96%)
Min alloc-odr0-512 213.00 ( 0.00%) 212.00 ( 0.47%)
Min alloc-odr0-1024 219.00 ( 0.00%) 219.00 ( 0.00%)
Min alloc-odr0-2048 225.00 ( 0.00%) 225.00 ( 0.00%)
Min alloc-odr0-4096 230.00 ( 0.00%) 231.00 ( -0.43%)
Min alloc-odr0-8192 235.00 ( 0.00%) 234.00 ( 0.43%)
Min alloc-odr0-16384 235.00 ( 0.00%) 234.00 ( 0.43%)
Min free-odr0-1 215.00 ( 0.00%) 191.00 ( 11.16%)
Min free-odr0-2 152.00 ( 0.00%) 136.00 ( 10.53%)
Min free-odr0-4 119.00 ( 0.00%) 107.00 ( 10.08%)
Min free-odr0-8 106.00 ( 0.00%) 96.00 ( 9.43%)
Min free-odr0-16 97.00 ( 0.00%) 87.00 ( 10.31%)
Min free-odr0-32 91.00 ( 0.00%) 83.00 ( 8.79%)
Min free-odr0-64 89.00 ( 0.00%) 81.00 ( 8.99%)
Min free-odr0-128 88.00 ( 0.00%) 80.00 ( 9.09%)
Min free-odr0-256 106.00 ( 0.00%) 95.00 ( 10.38%)
Min free-odr0-512 116.00 ( 0.00%) 111.00 ( 4.31%)
Min free-odr0-1024 125.00 ( 0.00%) 118.00 ( 5.60%)
Min free-odr0-2048 133.00 ( 0.00%) 126.00 ( 5.26%)
Min free-odr0-4096 136.00 ( 0.00%) 130.00 ( 4.41%)
Min free-odr0-8192 138.00 ( 0.00%) 130.00 ( 5.80%)
Min free-odr0-16384 137.00 ( 0.00%) 130.00 ( 5.11%)
There is a sizable boost to the free allocator performance. While there
is an apparent boost on the allocation side, it's likely a co-incidence
or due to the patches slightly reducing cache footprint.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
include/linux/page-flags.h | 7 ++++++-
mm/page_alloc.c | 2 +-
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index f4ed4f1b0c77..ccd04ee1ba2d 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -371,10 +371,15 @@ PAGEFLAG(Idle, idle, PF_ANY)
#define PAGE_MAPPING_KSM 2
#define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM)
+static __always_inline int PageAnonHead(struct page *page)
+{
+ return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0;
+}
+
static __always_inline int PageAnon(struct page *page)
{
page = compound_head(page);
- return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0;
+ return PageAnonHead(page);
}
#ifdef CONFIG_KSM
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 5d205bcfe10d..6812de41f698 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1048,7 +1048,7 @@ static bool free_pages_prepare(struct page *page, unsigned int order)
bad += free_pages_check(page + i);
}
}
- if (PageAnon(page))
+ if (PageAnonHead(page))
page->mapping = NULL;
bad += free_pages_check(page);
if (bad)
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-25 12:00 +0200 |
| Subject | Re: [PATCH 02/28] mm, page_alloc: Use new PageAnonHead helper in the free page fast path |
| Message-ID | <rrLt8-3Bp-11@gated-at.bofh.it> |
| In reply to | #1379576 |
On 04/15/2016 10:58 AM, Mel Gorman wrote:
> The PageAnon check always checks for compound_head but this is a relatively
> expensive check if the caller already knows the page is a head page. This
> patch creates a helper and uses it in the page free path which only operates
> on head pages.
>
> With this patch and "Only check PageCompound for high-order pages", the
> performance difference on a page allocator microbenchmark is;
>
[...]
>
> There is a sizable boost to the free allocator performance. While there
> is an apparent boost on the allocation side, it's likely a co-incidence
> or due to the patches slightly reducing cache footprint.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Acked-by: Vlastimil Babka <vbabka@suse.cz>
This again highlights the cost of thp rework due to those
compound_head() calls, and a more general solution would benefit other
places, but this can always be converted later if such solution happens.
> ---
> include/linux/page-flags.h | 7 ++++++-
> mm/page_alloc.c | 2 +-
> 2 files changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
> index f4ed4f1b0c77..ccd04ee1ba2d 100644
> --- a/include/linux/page-flags.h
> +++ b/include/linux/page-flags.h
> @@ -371,10 +371,15 @@ PAGEFLAG(Idle, idle, PF_ANY)
> #define PAGE_MAPPING_KSM 2
> #define PAGE_MAPPING_FLAGS (PAGE_MAPPING_ANON | PAGE_MAPPING_KSM)
>
> +static __always_inline int PageAnonHead(struct page *page)
> +{
> + return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0;
> +}
> +
> static __always_inline int PageAnon(struct page *page)
> {
> page = compound_head(page);
> - return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) != 0;
> + return PageAnonHead(page);
> }
>
> #ifdef CONFIG_KSM
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 5d205bcfe10d..6812de41f698 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -1048,7 +1048,7 @@ static bool free_pages_prepare(struct page *page, unsigned int order)
> bad += free_pages_check(page + i);
> }
> }
> - if (PageAnon(page))
> + if (PageAnonHead(page))
> page->mapping = NULL;
> bad += free_pages_check(page);
> if (bad)
>
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 11/28] mm, page_alloc: Remove unnecessary initialisation in get_page_from_freelist |
| Message-ID | <ro7Vh-Er-39@gated-at.bofh.it> |
| In reply to | #1379561 |
See subject.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/page_alloc.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 313db1c43839..f5ddb342c967 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2674,7 +2674,6 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
const struct alloc_context *ac)
{
struct zoneref *z;
- struct page *page = NULL;
struct zone *zone;
bool fair_skipped;
bool zonelist_rescan;
@@ -2688,6 +2687,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
*/
for_each_zone_zonelist_nodemask(zone, z, ac->zonelist, ac->high_zoneidx,
ac->nodemask) {
+ struct page *page;
unsigned long mark;
if (cpusets_enabled() &&
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 06/28] mm, page_alloc: Use __dec_zone_state for order-0 page allocation |
| Message-ID | <ro7Vh-Er-33@gated-at.bofh.it> |
| In reply to | #1379561 |
__dec_zone_state is cheaper to use for removing an order-0 page as it
has fewer conditions to check.
The performance difference on a page allocator microbenchmark is;
4.6.0-rc2 4.6.0-rc2
optiter-v1r20 decstat-v1r20
Min alloc-odr0-1 382.00 ( 0.00%) 381.00 ( 0.26%)
Min alloc-odr0-2 282.00 ( 0.00%) 275.00 ( 2.48%)
Min alloc-odr0-4 233.00 ( 0.00%) 229.00 ( 1.72%)
Min alloc-odr0-8 203.00 ( 0.00%) 199.00 ( 1.97%)
Min alloc-odr0-16 188.00 ( 0.00%) 186.00 ( 1.06%)
Min alloc-odr0-32 182.00 ( 0.00%) 179.00 ( 1.65%)
Min alloc-odr0-64 177.00 ( 0.00%) 174.00 ( 1.69%)
Min alloc-odr0-128 175.00 ( 0.00%) 172.00 ( 1.71%)
Min alloc-odr0-256 184.00 ( 0.00%) 181.00 ( 1.63%)
Min alloc-odr0-512 197.00 ( 0.00%) 193.00 ( 2.03%)
Min alloc-odr0-1024 203.00 ( 0.00%) 201.00 ( 0.99%)
Min alloc-odr0-2048 209.00 ( 0.00%) 206.00 ( 1.44%)
Min alloc-odr0-4096 214.00 ( 0.00%) 212.00 ( 0.93%)
Min alloc-odr0-8192 218.00 ( 0.00%) 215.00 ( 1.38%)
Min alloc-odr0-16384 219.00 ( 0.00%) 216.00 ( 1.37%)
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/page_alloc.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index e9acc0b0f787..ab16560b76e6 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2414,6 +2414,7 @@ struct page *buffered_rmqueue(struct zone *preferred_zone,
else
page = list_first_entry(list, struct page, lru);
+ __dec_zone_state(zone, NR_ALLOC_BATCH);
list_del(&page->lru);
pcp->count--;
} else {
@@ -2435,11 +2436,11 @@ struct page *buffered_rmqueue(struct zone *preferred_zone,
spin_unlock(&zone->lock);
if (!page)
goto failed;
+ __mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));
__mod_zone_freepage_state(zone, -(1 << order),
get_pcppage_migratetype(page));
}
- __mod_zone_page_state(zone, NR_ALLOC_BATCH, -(1 << order));
if (atomic_long_read(&zone->vm_stat[NR_ALLOC_BATCH]) <= 0 &&
!test_bit(ZONE_FAIR_DEPLETED, &zone->flags))
set_bit(ZONE_FAIR_DEPLETED, &zone->flags);
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-26 13:30 +0200 |
| Subject | Re: [PATCH 06/28] mm, page_alloc: Use __dec_zone_state for order-0 page allocation |
| Message-ID | <rs9lM-6K3-19@gated-at.bofh.it> |
| In reply to | #1379579 |
On 04/15/2016 10:58 AM, Mel Gorman wrote: > __dec_zone_state is cheaper to use for removing an order-0 page as it > has fewer conditions to check. > > The performance difference on a page allocator microbenchmark is; > > 4.6.0-rc2 4.6.0-rc2 > optiter-v1r20 decstat-v1r20 > Min alloc-odr0-1 382.00 ( 0.00%) 381.00 ( 0.26%) > Min alloc-odr0-2 282.00 ( 0.00%) 275.00 ( 2.48%) > Min alloc-odr0-4 233.00 ( 0.00%) 229.00 ( 1.72%) > Min alloc-odr0-8 203.00 ( 0.00%) 199.00 ( 1.97%) > Min alloc-odr0-16 188.00 ( 0.00%) 186.00 ( 1.06%) > Min alloc-odr0-32 182.00 ( 0.00%) 179.00 ( 1.65%) > Min alloc-odr0-64 177.00 ( 0.00%) 174.00 ( 1.69%) > Min alloc-odr0-128 175.00 ( 0.00%) 172.00 ( 1.71%) > Min alloc-odr0-256 184.00 ( 0.00%) 181.00 ( 1.63%) > Min alloc-odr0-512 197.00 ( 0.00%) 193.00 ( 2.03%) > Min alloc-odr0-1024 203.00 ( 0.00%) 201.00 ( 0.99%) > Min alloc-odr0-2048 209.00 ( 0.00%) 206.00 ( 1.44%) > Min alloc-odr0-4096 214.00 ( 0.00%) 212.00 ( 0.93%) > Min alloc-odr0-8192 218.00 ( 0.00%) 215.00 ( 1.38%) > Min alloc-odr0-16384 219.00 ( 0.00%) 216.00 ( 1.37%) > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Vlastimil Babka <vbabka@suse.cz>
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 09/28] mm, page_alloc: Convert nr_fair_skipped to bool |
| Message-ID | <ro7Vh-Er-35@gated-at.bofh.it> |
| In reply to | #1379561 |
The number of zones skipped to a zone expiring its fair zone allocation quota
is irrelevant. Convert to bool.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/page_alloc.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 4bce6298dd07..e778485a64c1 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2677,7 +2677,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
struct zoneref *z;
struct page *page = NULL;
struct zone *zone;
- int nr_fair_skipped = 0;
+ bool fair_skipped;
bool zonelist_rescan;
zonelist_scan:
@@ -2705,7 +2705,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
if (!zone_local(ac->preferred_zone, zone))
break;
if (test_bit(ZONE_FAIR_DEPLETED, &zone->flags)) {
- nr_fair_skipped++;
+ fair_skipped = true;
continue;
}
}
@@ -2798,7 +2798,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
*/
if (alloc_flags & ALLOC_FAIR) {
alloc_flags &= ~ALLOC_FAIR;
- if (nr_fair_skipped) {
+ if (fair_skipped) {
zonelist_rescan = true;
reset_alloc_batches(ac->preferred_zone);
}
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-26 13:40 +0200 |
| Subject | Re: [PATCH 09/28] mm, page_alloc: Convert nr_fair_skipped to bool |
| Message-ID | <rs9vr-6Of-3@gated-at.bofh.it> |
| In reply to | #1379580 |
On 04/15/2016 10:59 AM, Mel Gorman wrote: > The number of zones skipped to a zone expiring its fair zone allocation quota > is irrelevant. Convert to bool. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Vlastimil Babka <vbabka@suse.cz>
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 08/28] mm, page_alloc: Convert alloc_flags to unsigned |
| Message-ID | <ro7Vi-Er-47@gated-at.bofh.it> |
| In reply to | #1379561 |
alloc_flags is a bitmask of flags but it is signed which does not
necessarily generate the best code depending on the compiler. Even
without an impact, it makes more sense that this be unsigned.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
include/linux/compaction.h | 6 +++---
include/linux/mmzone.h | 3 ++-
mm/compaction.c | 12 +++++++-----
mm/internal.h | 2 +-
mm/page_alloc.c | 26 ++++++++++++++------------
5 files changed, 27 insertions(+), 22 deletions(-)
diff --git a/include/linux/compaction.h b/include/linux/compaction.h
index d7c8de583a23..242b660f64e6 100644
--- a/include/linux/compaction.h
+++ b/include/linux/compaction.h
@@ -39,12 +39,12 @@ extern int sysctl_compact_unevictable_allowed;
extern int fragmentation_index(struct zone *zone, unsigned int order);
extern unsigned long try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
- int alloc_flags, const struct alloc_context *ac,
- enum migrate_mode mode, int *contended);
+ unsigned int alloc_flags, const struct alloc_context *ac,
+ enum migrate_mode mode, int *contended);
extern void compact_pgdat(pg_data_t *pgdat, int order);
extern void reset_isolation_suitable(pg_data_t *pgdat);
extern unsigned long compaction_suitable(struct zone *zone, int order,
- int alloc_flags, int classzone_idx);
+ unsigned int alloc_flags, int classzone_idx);
extern void defer_compaction(struct zone *zone, int order);
extern bool compaction_deferred(struct zone *zone, int order);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 0c4d5ebb3849..f49bb9add372 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -747,7 +747,8 @@ extern struct mutex zonelists_mutex;
void build_all_zonelists(pg_data_t *pgdat, struct zone *zone);
void wakeup_kswapd(struct zone *zone, int order, enum zone_type classzone_idx);
bool zone_watermark_ok(struct zone *z, unsigned int order,
- unsigned long mark, int classzone_idx, int alloc_flags);
+ unsigned long mark, int classzone_idx,
+ unsigned int alloc_flags);
bool zone_watermark_ok_safe(struct zone *z, unsigned int order,
unsigned long mark, int classzone_idx);
enum memmap_context {
diff --git a/mm/compaction.c b/mm/compaction.c
index ccf97b02b85f..244bb669b5a6 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -1259,7 +1259,8 @@ static int compact_finished(struct zone *zone, struct compact_control *cc,
* COMPACT_CONTINUE - If compaction should run now
*/
static unsigned long __compaction_suitable(struct zone *zone, int order,
- int alloc_flags, int classzone_idx)
+ unsigned int alloc_flags,
+ int classzone_idx)
{
int fragindex;
unsigned long watermark;
@@ -1304,7 +1305,8 @@ static unsigned long __compaction_suitable(struct zone *zone, int order,
}
unsigned long compaction_suitable(struct zone *zone, int order,
- int alloc_flags, int classzone_idx)
+ unsigned int alloc_flags,
+ int classzone_idx)
{
unsigned long ret;
@@ -1464,7 +1466,7 @@ static int compact_zone(struct zone *zone, struct compact_control *cc)
static unsigned long compact_zone_order(struct zone *zone, int order,
gfp_t gfp_mask, enum migrate_mode mode, int *contended,
- int alloc_flags, int classzone_idx)
+ unsigned int alloc_flags, int classzone_idx)
{
unsigned long ret;
struct compact_control cc = {
@@ -1505,8 +1507,8 @@ int sysctl_extfrag_threshold = 500;
* This is the main entry point for direct page compaction.
*/
unsigned long try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
- int alloc_flags, const struct alloc_context *ac,
- enum migrate_mode mode, int *contended)
+ unsigned int alloc_flags, const struct alloc_context *ac,
+ enum migrate_mode mode, int *contended)
{
int may_enter_fs = gfp_mask & __GFP_FS;
int may_perform_io = gfp_mask & __GFP_IO;
diff --git a/mm/internal.h b/mm/internal.h
index b79abb6721cf..f6d0a5875ec4 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -175,7 +175,7 @@ struct compact_control {
bool direct_compaction; /* False from kcompactd or /proc/... */
int order; /* order a direct compactor needs */
const gfp_t gfp_mask; /* gfp mask of a direct compactor */
- const int alloc_flags; /* alloc flags of a direct compactor */
+ const unsigned int alloc_flags; /* alloc flags of a direct compactor */
const int classzone_idx; /* zone index of a direct compactor */
struct zone *zone;
int contended; /* Signal need_sched() or lock
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index d00847bb1612..4bce6298dd07 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -1526,7 +1526,7 @@ static inline bool free_pages_prezeroed(bool poisoned)
}
static int prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags,
- int alloc_flags)
+ unsigned int alloc_flags)
{
int i;
bool poisoned = true;
@@ -2388,7 +2388,8 @@ static inline void zone_statistics(struct zone *preferred_zone, struct zone *z,
static inline
struct page *buffered_rmqueue(struct zone *preferred_zone,
struct zone *zone, unsigned int order,
- gfp_t gfp_flags, int alloc_flags, int migratetype)
+ gfp_t gfp_flags, unsigned int alloc_flags,
+ int migratetype)
{
unsigned long flags;
struct page *page;
@@ -2542,12 +2543,13 @@ static inline bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
* to check in the allocation paths if no pages are free.
*/
static bool __zone_watermark_ok(struct zone *z, unsigned int order,
- unsigned long mark, int classzone_idx, int alloc_flags,
+ unsigned long mark, int classzone_idx,
+ unsigned int alloc_flags,
long free_pages)
{
long min = mark;
int o;
- const int alloc_harder = (alloc_flags & ALLOC_HARDER);
+ const bool alloc_harder = (alloc_flags & ALLOC_HARDER);
/* free_pages may go negative - that's OK */
free_pages -= (1 << order) - 1;
@@ -2610,7 +2612,7 @@ static bool __zone_watermark_ok(struct zone *z, unsigned int order,
}
bool zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
- int classzone_idx, int alloc_flags)
+ int classzone_idx, unsigned int alloc_flags)
{
return __zone_watermark_ok(z, order, mark, classzone_idx, alloc_flags,
zone_page_state(z, NR_FREE_PAGES));
@@ -2958,7 +2960,7 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,
/* Try memory compaction for high-order allocations before reclaim */
static struct page *
__alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
- int alloc_flags, const struct alloc_context *ac,
+ unsigned int alloc_flags, const struct alloc_context *ac,
enum migrate_mode mode, int *contended_compaction,
bool *deferred_compaction)
{
@@ -3014,7 +3016,7 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
#else
static inline struct page *
__alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
- int alloc_flags, const struct alloc_context *ac,
+ unsigned int alloc_flags, const struct alloc_context *ac,
enum migrate_mode mode, int *contended_compaction,
bool *deferred_compaction)
{
@@ -3054,7 +3056,7 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,
/* The really slow allocator path where we enter direct reclaim */
static inline struct page *
__alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,
- int alloc_flags, const struct alloc_context *ac,
+ unsigned int alloc_flags, const struct alloc_context *ac,
unsigned long *did_some_progress)
{
struct page *page = NULL;
@@ -3093,10 +3095,10 @@ static void wake_all_kswapds(unsigned int order, const struct alloc_context *ac)
wakeup_kswapd(zone, order, zone_idx(ac->preferred_zone));
}
-static inline int
+static inline unsigned int
gfp_to_alloc_flags(gfp_t gfp_mask)
{
- int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET;
+ unsigned int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET;
/* __GFP_HIGH is assumed to be the same as ALLOC_HIGH to save a branch. */
BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_HIGH);
@@ -3157,7 +3159,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
{
bool can_direct_reclaim = gfp_mask & __GFP_DIRECT_RECLAIM;
struct page *page = NULL;
- int alloc_flags;
+ unsigned int alloc_flags;
unsigned long pages_reclaimed = 0;
unsigned long did_some_progress;
enum migrate_mode migration_mode = MIGRATE_ASYNC;
@@ -3349,7 +3351,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
struct zoneref *preferred_zoneref;
struct page *page = NULL;
unsigned int cpuset_mems_cookie;
- int alloc_flags = ALLOC_WMARK_LOW|ALLOC_FAIR;
+ unsigned int alloc_flags = ALLOC_WMARK_LOW|ALLOC_FAIR;
gfp_t alloc_mask; /* The gfp_t that was actually used for allocation */
struct alloc_context ac = {
.high_zoneidx = gfp_zone(gfp_mask),
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2016-04-15 11:10 +0200 |
| Subject | [PATCH 03/28] mm, page_alloc: Reduce branches in zone_statistics |
| Message-ID | <ro7Vi-Er-53@gated-at.bofh.it> |
| In reply to | #1379561 |
zone_statistics has more branches than it really needs to take an
unlikely GFP flag into account. Reduce the number and annotate
the unlikely flag.
The performance difference on a page allocator microbenchmark is;
4.6.0-rc2 4.6.0-rc2
nocompound-v1r10 statbranch-v1r10
Min alloc-odr0-1 417.00 ( 0.00%) 419.00 ( -0.48%)
Min alloc-odr0-2 308.00 ( 0.00%) 305.00 ( 0.97%)
Min alloc-odr0-4 253.00 ( 0.00%) 250.00 ( 1.19%)
Min alloc-odr0-8 221.00 ( 0.00%) 219.00 ( 0.90%)
Min alloc-odr0-16 205.00 ( 0.00%) 203.00 ( 0.98%)
Min alloc-odr0-32 199.00 ( 0.00%) 195.00 ( 2.01%)
Min alloc-odr0-64 193.00 ( 0.00%) 191.00 ( 1.04%)
Min alloc-odr0-128 191.00 ( 0.00%) 189.00 ( 1.05%)
Min alloc-odr0-256 200.00 ( 0.00%) 198.00 ( 1.00%)
Min alloc-odr0-512 212.00 ( 0.00%) 210.00 ( 0.94%)
Min alloc-odr0-1024 219.00 ( 0.00%) 216.00 ( 1.37%)
Min alloc-odr0-2048 225.00 ( 0.00%) 221.00 ( 1.78%)
Min alloc-odr0-4096 231.00 ( 0.00%) 227.00 ( 1.73%)
Min alloc-odr0-8192 234.00 ( 0.00%) 232.00 ( 0.85%)
Min alloc-odr0-16384 234.00 ( 0.00%) 232.00 ( 0.85%)
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/vmstat.c | 16 ++++++++++------
1 file changed, 10 insertions(+), 6 deletions(-)
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 5e4300482897..2e58ead9bcf5 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -581,17 +581,21 @@ void drain_zonestat(struct zone *zone, struct per_cpu_pageset *pset)
*/
void zone_statistics(struct zone *preferred_zone, struct zone *z, gfp_t flags)
{
- if (z->zone_pgdat == preferred_zone->zone_pgdat) {
+ int local_nid = numa_node_id();
+ enum zone_stat_item local_stat = NUMA_LOCAL;
+
+ if (unlikely(flags & __GFP_OTHER_NODE)) {
+ local_stat = NUMA_OTHER;
+ local_nid = preferred_zone->node;
+ }
+
+ if (z->node == local_nid) {
__inc_zone_state(z, NUMA_HIT);
+ __inc_zone_state(z, local_stat);
} else {
__inc_zone_state(z, NUMA_MISS);
__inc_zone_state(preferred_zone, NUMA_FOREIGN);
}
- if (z->node == ((flags & __GFP_OTHER_NODE) ?
- preferred_zone->node : numa_node_id()))
- __inc_zone_state(z, NUMA_LOCAL);
- else
- __inc_zone_state(z, NUMA_OTHER);
}
/*
--
2.6.4
[toc] | [prev] | [next] | [standalone]
| From | Vlastimil Babka <vbabka@suse.cz> |
|---|---|
| Date | 2016-04-25 13:20 +0200 |
| Subject | Re: [PATCH 03/28] mm, page_alloc: Reduce branches in zone_statistics |
| Message-ID | <rrMIx-4QJ-13@gated-at.bofh.it> |
| In reply to | #1379584 |
On 04/15/2016 10:58 AM, Mel Gorman wrote: > zone_statistics has more branches than it really needs to take an > unlikely GFP flag into account. Reduce the number and annotate > the unlikely flag. > > The performance difference on a page allocator microbenchmark is; > > 4.6.0-rc2 4.6.0-rc2 > nocompound-v1r10 statbranch-v1r10 > Min alloc-odr0-1 417.00 ( 0.00%) 419.00 ( -0.48%) > Min alloc-odr0-2 308.00 ( 0.00%) 305.00 ( 0.97%) > Min alloc-odr0-4 253.00 ( 0.00%) 250.00 ( 1.19%) > Min alloc-odr0-8 221.00 ( 0.00%) 219.00 ( 0.90%) > Min alloc-odr0-16 205.00 ( 0.00%) 203.00 ( 0.98%) > Min alloc-odr0-32 199.00 ( 0.00%) 195.00 ( 2.01%) > Min alloc-odr0-64 193.00 ( 0.00%) 191.00 ( 1.04%) > Min alloc-odr0-128 191.00 ( 0.00%) 189.00 ( 1.05%) > Min alloc-odr0-256 200.00 ( 0.00%) 198.00 ( 1.00%) > Min alloc-odr0-512 212.00 ( 0.00%) 210.00 ( 0.94%) > Min alloc-odr0-1024 219.00 ( 0.00%) 216.00 ( 1.37%) > Min alloc-odr0-2048 225.00 ( 0.00%) 221.00 ( 1.78%) > Min alloc-odr0-4096 231.00 ( 0.00%) 227.00 ( 1.73%) > Min alloc-odr0-8192 234.00 ( 0.00%) 232.00 ( 0.85%) > Min alloc-odr0-16384 234.00 ( 0.00%) 232.00 ( 0.85%) > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Vlastimil Babka <vbabka@suse.cz>
[toc] | [prev] | [next] | [standalone]
Page 3 of 4 — ← Prev page 1 2 [3] 4 Next page →
Back to top | Article view | linux.kernel
csiph-web