Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1554471 > unrolled thread
| Started by | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| First post | 2017-01-09 17:40 +0100 |
| Last post | 2017-01-12 04:20 +0100 |
| Articles | 4 — 3 participants |
Back to article view | Back to linux.kernel
[RFC PATCH 0/4] Fast noirq bulk page allocator v2r7 Mel Gorman <mgorman@techsingularity.net> - 2017-01-09 17:40 +0100
[PATCH 2/4] mm, page_alloc: Split alloc_pages_nodemask Mel Gorman <mgorman@techsingularity.net> - 2017-01-09 17:40 +0100
Re: [PATCH 2/4] mm, page_alloc: Split alloc_pages_nodemask Jesper Dangaard Brouer <brouer@redhat.com> - 2017-01-11 13:40 +0100
Re: [PATCH 2/4] mm, page_alloc: Split alloc_pages_nodemask "Hillf Danton" <hillf.zj@alibaba-inc.com> - 2017-01-12 04:20 +0100
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2017-01-09 17:40 +0100 |
| Subject | [RFC PATCH 0/4] Fast noirq bulk page allocator v2r7 |
| Message-ID | <sXL9f-6vS-3@gated-at.bofh.it> |
The biggest changes are in the final patch. In v1, it was a rough untested
prototype. This version corrected a number of issues, tested it and includes
a comparison between bulk allocating pages and allocating them one at a time.
While there are still no in-kernel users, it is hoped that the bulk API
would convince network drivers to avoid using high-order allocations. One
slight caveat is that there still may be an advantage to doing the coherent
setup on a high-order page instead of a list of order-0 pages. If that is the
case, it would need to be covered by Jesper's generic page pool allocator.
Changelog since v1
o Remove a scheduler point from the allocation path
o Finalise the bulk allocator and test it
This series is motivated by a conversation led by Jesper Dangaard Brouer at
the last LSF/MM proposing a generic page pool for DMA-coherent pages. Part of
his motivation was due to the overhead of allocating multiple order-0 that
led some drivers to use high-order allocations and splitting them which
can be very slow if high-order pages are unavailable. This long-overdue
series aims to show that raw bulk page allocation can be achieved relatively
easily without introducing a completely new allocator. A new generic page
pool allocator would then ideally focus on just the DMA-coherent part.
The first two patches in the series restructure the allocator such that
it's relatively easy to build a bulk page allocator. The third patch
alters the per-cpu alloctor to make it exclusive to !irq requests. This
cuts allocation/free overhead by roughly 30% but it may not be noticable
to anyone other than users of high-speed networks (I'm not one). The
fourth patch introduces a bulk page allocator with no in-kernel users as
an example for Jesper and others who want to build a page allocator for
DMA-coherent pages. It hopefully is relatively easy to modify this API
and the one core function to get the semantics they require. Note that
Patch 3 is not required for patch 4 but it may be desirable if the bulk
allocations happen from !IRQ context.
A comparison of costs of allocating one page at a time on the vanilla
kernel vs the bulk allocator that forces the per-cpu allocator to be
used from a !irq context is as follows
pagealloc
4.10.0-rc2 4.10.0-rc2
vanilla bulk-v2r7
Amean alloc-odr0-1 302.85 ( 0.00%) 106.62 ( 64.80%)
Amean alloc-odr0-2 227.85 ( 0.00%) 76.38 ( 66.48%)
Amean alloc-odr0-4 191.23 ( 0.00%) 57.23 ( 70.07%)
Amean alloc-odr0-8 167.54 ( 0.00%) 48.77 ( 70.89%)
Amean alloc-odr0-16 158.54 ( 0.00%) 45.38 ( 71.37%)
Amean alloc-odr0-32 150.46 ( 0.00%) 42.77 ( 71.57%)
Amean alloc-odr0-64 148.23 ( 0.00%) 41.00 ( 72.34%)
Amean alloc-odr0-128 145.00 ( 0.00%) 40.08 ( 72.36%)
Amean alloc-odr0-256 157.00 ( 0.00%) 56.00 ( 64.33%)
Amean alloc-odr0-512 170.00 ( 0.00%) 69.00 ( 59.41%)
Amean alloc-odr0-1024 181.00 ( 0.00%) 76.23 ( 57.88%)
Amean alloc-odr0-2048 186.00 ( 0.00%) 81.15 ( 56.37%)
Amean alloc-odr0-4096 192.92 ( 0.00%) 85.92 ( 55.46%)
Amean alloc-odr0-8192 194.00 ( 0.00%) 88.00 ( 54.64%)
Amean alloc-odr0-16384 202.15 ( 0.00%) 89.00 ( 55.97%)
Amean free-odr0-1 154.92 ( 0.00%) 55.69 ( 64.05%)
Amean free-odr0-2 115.31 ( 0.00%) 49.38 ( 57.17%)
Amean free-odr0-4 93.31 ( 0.00%) 45.38 ( 51.36%)
Amean free-odr0-8 82.62 ( 0.00%) 44.23 ( 46.46%)
Amean free-odr0-16 79.00 ( 0.00%) 45.00 ( 43.04%)
Amean free-odr0-32 75.15 ( 0.00%) 43.92 ( 41.56%)
Amean free-odr0-64 74.00 ( 0.00%) 43.00 ( 41.89%)
Amean free-odr0-128 73.00 ( 0.00%) 43.00 ( 41.10%)
Amean free-odr0-256 91.00 ( 0.00%) 60.46 ( 33.56%)
Amean free-odr0-512 108.00 ( 0.00%) 76.00 ( 29.63%)
Amean free-odr0-1024 119.00 ( 0.00%) 85.38 ( 28.25%)
Amean free-odr0-2048 125.08 ( 0.00%) 91.23 ( 27.06%)
Amean free-odr0-4096 130.00 ( 0.00%) 95.62 ( 26.45%)
Amean free-odr0-8192 130.00 ( 0.00%) 97.00 ( 25.38%)
Amean free-odr0-16384 134.46 ( 0.00%) 97.46 ( 27.52%)
Amean total-odr0-1 457.77 ( 0.00%) 162.31 ( 64.54%)
Amean total-odr0-2 343.15 ( 0.00%) 125.77 ( 63.35%)
Amean total-odr0-4 284.54 ( 0.00%) 102.62 ( 63.94%)
Amean total-odr0-8 250.15 ( 0.00%) 93.00 ( 62.82%)
Amean total-odr0-16 237.54 ( 0.00%) 90.38 ( 61.95%)
Amean total-odr0-32 225.62 ( 0.00%) 86.69 ( 61.58%)
Amean total-odr0-64 222.23 ( 0.00%) 84.00 ( 62.20%)
Amean total-odr0-128 218.00 ( 0.00%) 83.08 ( 61.89%)
Amean total-odr0-256 248.00 ( 0.00%) 116.46 ( 53.04%)
Amean total-odr0-512 278.00 ( 0.00%) 145.00 ( 47.84%)
Amean total-odr0-1024 300.00 ( 0.00%) 161.62 ( 46.13%)
Amean total-odr0-2048 311.08 ( 0.00%) 172.38 ( 44.58%)
Amean total-odr0-4096 322.92 ( 0.00%) 181.54 ( 43.78%)
Amean total-odr0-8192 324.00 ( 0.00%) 185.00 ( 42.90%)
Amean total-odr0-16384 336.62 ( 0.00%) 186.46 ( 44.61%)
It's roughly a 50-70% reduction of allocation costs and roughly a halving of the
overall cost of allocating/freeing batches of pages.
include/linux/gfp.h | 24 ++++
mm/page_alloc.c | 353 +++++++++++++++++++++++++++++++++++++---------------
2 files changed, 278 insertions(+), 99 deletions(-)
--
2.11.0
[toc] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2017-01-09 17:40 +0100 |
| Subject | [PATCH 2/4] mm, page_alloc: Split alloc_pages_nodemask |
| Message-ID | <sXL9g-6vS-35@gated-at.bofh.it> |
| In reply to | #1554471 |
alloc_pages_nodemask does a number of preperation steps that determine
what zones can be used for the allocation depending on a variety of
factors. This is fine but a hypothetical caller that wanted multiple
order-0 pages has to do the preparation steps multiple times. This patch
structures __alloc_pages_nodemask such that it's relatively easy to build
a bulk order-0 page allocator. There is no functional change.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
mm/page_alloc.c | 81 ++++++++++++++++++++++++++++++++++-----------------------
1 file changed, 49 insertions(+), 32 deletions(-)
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index d8798583eaf8..4a602b7f258d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3762,64 +3762,81 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
return page;
}
-/*
- * This is the 'heart' of the zoned buddy allocator.
- */
-struct page *
-__alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
- struct zonelist *zonelist, nodemask_t *nodemask)
+static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
+ struct zonelist *zonelist, nodemask_t *nodemask,
+ struct alloc_context *ac, gfp_t *alloc_mask,
+ unsigned int *alloc_flags)
{
- struct page *page;
- unsigned int cpuset_mems_cookie;
- unsigned int alloc_flags = ALLOC_WMARK_LOW;
- gfp_t alloc_mask = gfp_mask; /* The gfp_t that was actually used for allocation */
- struct alloc_context ac = {
- .high_zoneidx = gfp_zone(gfp_mask),
- .zonelist = zonelist,
- .nodemask = nodemask,
- .migratetype = gfpflags_to_migratetype(gfp_mask),
- };
+ ac->high_zoneidx = gfp_zone(gfp_mask);
+ ac->zonelist = zonelist;
+ ac->nodemask = nodemask;
+ ac->migratetype = gfpflags_to_migratetype(gfp_mask);
if (cpusets_enabled()) {
- alloc_mask |= __GFP_HARDWALL;
- alloc_flags |= ALLOC_CPUSET;
- if (!ac.nodemask)
- ac.nodemask = &cpuset_current_mems_allowed;
+ *alloc_mask |= __GFP_HARDWALL;
+ *alloc_flags |= ALLOC_CPUSET;
+ if (!ac->nodemask)
+ ac->nodemask = &cpuset_current_mems_allowed;
}
- gfp_mask &= gfp_allowed_mask;
-
lockdep_trace_alloc(gfp_mask);
might_sleep_if(gfp_mask & __GFP_DIRECT_RECLAIM);
if (should_fail_alloc_page(gfp_mask, order))
- return NULL;
+ return false;
/*
* Check the zones suitable for the gfp_mask contain at least one
* valid zone. It's possible to have an empty zonelist as a result
* of __GFP_THISNODE and a memoryless node
*/
- if (unlikely(!zonelist->_zonerefs->zone))
- return NULL;
+ if (unlikely(!ac->zonelist->_zonerefs->zone))
+ return false;
- if (IS_ENABLED(CONFIG_CMA) && ac.migratetype == MIGRATE_MOVABLE)
- alloc_flags |= ALLOC_CMA;
+ if (IS_ENABLED(CONFIG_CMA) && ac->migratetype == MIGRATE_MOVABLE)
+ *alloc_flags |= ALLOC_CMA;
-retry_cpuset:
- cpuset_mems_cookie = read_mems_allowed_begin();
+ return true;
+}
+/* Determine whether to spread dirty pages and what the first usable zone */
+static inline void finalise_ac(gfp_t gfp_mask,
+ unsigned int order, struct alloc_context *ac)
+{
/* Dirty zone balancing only done in the fast path */
- ac.spread_dirty_pages = (gfp_mask & __GFP_WRITE);
+ ac->spread_dirty_pages = (gfp_mask & __GFP_WRITE);
/*
* The preferred zone is used for statistics but crucially it is
* also used as the starting point for the zonelist iterator. It
* may get reset for allocations that ignore memory policies.
*/
- ac.preferred_zoneref = first_zones_zonelist(ac.zonelist,
- ac.high_zoneidx, ac.nodemask);
+ ac->preferred_zoneref = first_zones_zonelist(ac->zonelist,
+ ac->high_zoneidx, ac->nodemask);
+}
+
+/*
+ * This is the 'heart' of the zoned buddy allocator.
+ */
+struct page *
+__alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
+ struct zonelist *zonelist, nodemask_t *nodemask)
+{
+ struct page *page;
+ unsigned int cpuset_mems_cookie;
+ unsigned int alloc_flags = ALLOC_WMARK_LOW;
+ gfp_t alloc_mask = gfp_mask; /* The gfp_t that was actually used for allocation */
+ struct alloc_context ac = { };
+
+ gfp_mask &= gfp_allowed_mask;
+ if (!prepare_alloc_pages(gfp_mask, order, zonelist, nodemask, &ac, &alloc_mask, &alloc_flags))
+ return NULL;
+
+retry_cpuset:
+ cpuset_mems_cookie = read_mems_allowed_begin();
+
+ finalise_ac(gfp_mask, order, &ac);
if (!ac.preferred_zoneref) {
page = NULL;
goto no_zone;
--
2.11.0
[toc] | [prev] | [next] | [standalone]
| From | Jesper Dangaard Brouer <brouer@redhat.com> |
|---|---|
| Date | 2017-01-11 13:40 +0100 |
| Subject | Re: [PATCH 2/4] mm, page_alloc: Split alloc_pages_nodemask |
| Message-ID | <sYqm6-6WF-9@gated-at.bofh.it> |
| In reply to | #1554472 |
On Mon, 9 Jan 2017 16:35:16 +0000 Mel Gorman <mgorman@techsingularity.net> wrote: > alloc_pages_nodemask does a number of preperation steps that determine > what zones can be used for the allocation depending on a variety of > factors. This is fine but a hypothetical caller that wanted multiple > order-0 pages has to do the preparation steps multiple times. This patch > structures __alloc_pages_nodemask such that it's relatively easy to build > a bulk order-0 page allocator. There is no functional change. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
[toc] | [prev] | [next] | [standalone]
| From | "Hillf Danton" <hillf.zj@alibaba-inc.com> |
|---|---|
| Date | 2017-01-12 04:20 +0100 |
| Subject | Re: [PATCH 2/4] mm, page_alloc: Split alloc_pages_nodemask |
| Message-ID | <sYE5I-78y-9@gated-at.bofh.it> |
| In reply to | #1554472 |
On Tuesday, January 10, 2017 12:35 AM Mel Gorman wrote: > > alloc_pages_nodemask does a number of preperation steps that determine > what zones can be used for the allocation depending on a variety of > factors. This is fine but a hypothetical caller that wanted multiple > order-0 pages has to do the preparation steps multiple times. This patch > structures __alloc_pages_nodemask such that it's relatively easy to build > a bulk order-0 page allocator. There is no functional change. > > Signed-off-by: Mel Gorman <mgorman@techsingularity.net> > --- Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com>
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web