Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1554464 > unrolled thread
| Started by | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| First post | 2017-01-09 17:40 +0100 |
| Last post | 2017-01-16 16:10 +0100 |
| Articles | 5 — 3 participants |
Back to article view | Back to linux.kernel
This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by
below is the oldest one visible, not the original post.
[PATCH 4/4] mm, page_alloc: Add a bulk page allocator Mel Gorman <mgorman@techsingularity.net> - 2017-01-09 17:40 +0100
Re: [PATCH 4/4] mm, page_alloc: Add a bulk page allocator "Hillf Danton" <hillf.zj@alibaba-inc.com> - 2017-01-10 05:10 +0100
Re: [PATCH 4/4] mm, page_alloc: Add a bulk page allocator Mel Gorman <mgorman@techsingularity.net> - 2017-01-10 09:40 +0100
Re: [PATCH 4/4] mm, page_alloc: Add a bulk page allocator Jesper Dangaard Brouer <brouer@redhat.com> - 2017-01-16 15:30 +0100
Re: [PATCH 4/4] mm, page_alloc: Add a bulk page allocator Mel Gorman <mgorman@techsingularity.net> - 2017-01-16 16:10 +0100
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2017-01-09 17:40 +0100 |
| Subject | [PATCH 4/4] mm, page_alloc: Add a bulk page allocator |
| Message-ID | <sXL9f-6vS-1@gated-at.bofh.it> |
This patch adds a new page allocator interface via alloc_pages_bulk,
__alloc_pages_bulk and __alloc_pages_bulk_nodemask. A caller requests a
number of pages to be allocated and added to a list. They can be freed in
bulk using free_pages_bulk(). Note that it would theoretically be possible
to use free_hot_cold_page_list for faster frees if the symbol was exported,
the refcounts were 0 and the caller guaranteed it was not in an interrupt.
This would be significantly faster in the free path but also more unsafer
and a harder API to use.
The API is not guaranteed to return the requested number of pages and
may fail if the preferred allocation zone has limited free memory, the
cpuset changes during the allocation or page debugging decides to fail
an allocation. It's up to the caller to request more pages in batch if
necessary.
The following compares the allocation cost per page for different batch
sizes. The baseline is allocating them one at a time and it compares with
the performance when using the new allocation interface.
pagealloc
4.10.0-rc2 4.10.0-rc2
one-at-a-time bulk-v2
Amean alloc-odr0-1 259.54 ( 0.00%) 106.62 ( 58.92%)
Amean alloc-odr0-2 193.38 ( 0.00%) 76.38 ( 60.50%)
Amean alloc-odr0-4 162.38 ( 0.00%) 57.23 ( 64.76%)
Amean alloc-odr0-8 144.31 ( 0.00%) 48.77 ( 66.20%)
Amean alloc-odr0-16 134.08 ( 0.00%) 45.38 ( 66.15%)
Amean alloc-odr0-32 128.62 ( 0.00%) 42.77 ( 66.75%)
Amean alloc-odr0-64 126.00 ( 0.00%) 41.00 ( 67.46%)
Amean alloc-odr0-128 125.00 ( 0.00%) 40.08 ( 67.94%)
Amean alloc-odr0-256 136.62 ( 0.00%) 56.00 ( 59.01%)
Amean alloc-odr0-512 152.00 ( 0.00%) 69.00 ( 54.61%)
Amean alloc-odr0-1024 158.00 ( 0.00%) 76.23 ( 51.75%)
Amean alloc-odr0-2048 163.00 ( 0.00%) 81.15 ( 50.21%)
Amean alloc-odr0-4096 169.77 ( 0.00%) 85.92 ( 49.39%)
Amean alloc-odr0-8192 170.00 ( 0.00%) 88.00 ( 48.24%)
Amean alloc-odr0-16384 170.00 ( 0.00%) 89.00 ( 47.65%)
Amean free-odr0-1 88.69 ( 0.00%) 55.69 ( 37.21%)
Amean free-odr0-2 66.00 ( 0.00%) 49.38 ( 25.17%)
Amean free-odr0-4 54.23 ( 0.00%) 45.38 ( 16.31%)
Amean free-odr0-8 48.23 ( 0.00%) 44.23 ( 8.29%)
Amean free-odr0-16 47.00 ( 0.00%) 45.00 ( 4.26%)
Amean free-odr0-32 44.77 ( 0.00%) 43.92 ( 1.89%)
Amean free-odr0-64 44.00 ( 0.00%) 43.00 ( 2.27%)
Amean free-odr0-128 43.00 ( 0.00%) 43.00 ( 0.00%)
Amean free-odr0-256 60.69 ( 0.00%) 60.46 ( 0.38%)
Amean free-odr0-512 79.23 ( 0.00%) 76.00 ( 4.08%)
Amean free-odr0-1024 86.00 ( 0.00%) 85.38 ( 0.72%)
Amean free-odr0-2048 91.00 ( 0.00%) 91.23 ( -0.25%)
Amean free-odr0-4096 94.85 ( 0.00%) 95.62 ( -0.81%)
Amean free-odr0-8192 97.00 ( 0.00%) 97.00 ( 0.00%)
Amean free-odr0-16384 98.00 ( 0.00%) 97.46 ( 0.55%)
Amean total-odr0-1 348.23 ( 0.00%) 162.31 ( 53.39%)
Amean total-odr0-2 259.38 ( 0.00%) 125.77 ( 51.51%)
Amean total-odr0-4 216.62 ( 0.00%) 102.62 ( 52.63%)
Amean total-odr0-8 192.54 ( 0.00%) 93.00 ( 51.70%)
Amean total-odr0-16 181.08 ( 0.00%) 90.38 ( 50.08%)
Amean total-odr0-32 173.38 ( 0.00%) 86.69 ( 50.00%)
Amean total-odr0-64 170.00 ( 0.00%) 84.00 ( 50.59%)
Amean total-odr0-128 168.00 ( 0.00%) 83.08 ( 50.55%)
Amean total-odr0-256 197.31 ( 0.00%) 116.46 ( 40.97%)
Amean total-odr0-512 231.23 ( 0.00%) 145.00 ( 37.29%)
Amean total-odr0-1024 244.00 ( 0.00%) 161.62 ( 33.76%)
Amean total-odr0-2048 254.00 ( 0.00%) 172.38 ( 32.13%)
Amean total-odr0-4096 264.62 ( 0.00%) 181.54 ( 31.40%)
Amean total-odr0-8192 267.00 ( 0.00%) 185.00 ( 30.71%)
Amean total-odr0-16384 268.00 ( 0.00%) 186.46 ( 30.42%)
It shows a roughly 50-60% reduction in the cost of allocating pages.
The free paths are not improved as much but relatively little can be batched
there. It's not quite as fast as it could be but taking further shortcuts
would require making a lot of assumptions about the state of the page and
the context of the caller.
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
---
include/linux/gfp.h | 24 +++++++++++
mm/page_alloc.c | 116 +++++++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 139 insertions(+), 1 deletion(-)
diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 4175dca4ac39..b2fe171ee1c4 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -433,6 +433,29 @@ __alloc_pages(gfp_t gfp_mask, unsigned int order,
return __alloc_pages_nodemask(gfp_mask, order, zonelist, NULL);
}
+unsigned long
+__alloc_pages_bulk_nodemask(gfp_t gfp_mask, unsigned int order,
+ struct zonelist *zonelist, nodemask_t *nodemask,
+ unsigned long nr_pages, struct list_head *alloc_list);
+
+static inline unsigned long
+__alloc_pages_bulk(gfp_t gfp_mask, unsigned int order,
+ struct zonelist *zonelist, unsigned long nr_pages,
+ struct list_head *list)
+{
+ return __alloc_pages_bulk_nodemask(gfp_mask, order, zonelist, NULL,
+ nr_pages, list);
+}
+
+static inline unsigned long
+alloc_pages_bulk(gfp_t gfp_mask, unsigned int order,
+ unsigned long nr_pages, struct list_head *list)
+{
+ int nid = numa_mem_id();
+ return __alloc_pages_bulk(gfp_mask, order,
+ node_zonelist(nid, gfp_mask), nr_pages, list);
+}
+
/*
* Allocate pages, preferring the node given as nid. The node must be valid and
* online. For more general interface, see alloc_pages_node().
@@ -504,6 +527,7 @@ extern void __free_pages(struct page *page, unsigned int order);
extern void free_pages(unsigned long addr, unsigned int order);
extern void free_hot_cold_page(struct page *page, bool cold);
extern void free_hot_cold_page_list(struct list_head *list, bool cold);
+extern void free_pages_bulk(struct list_head *list);
struct page_frag_cache;
extern void __page_frag_drain(struct page *page, unsigned int order,
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 232cadbe9231..4f142270fbf0 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -2485,7 +2485,7 @@ void free_hot_cold_page(struct page *page, bool cold)
}
/*
- * Free a list of 0-order pages
+ * Free a list of 0-order pages whose reference count is already zero.
*/
void free_hot_cold_page_list(struct list_head *list, bool cold)
{
@@ -2495,7 +2495,28 @@ void free_hot_cold_page_list(struct list_head *list, bool cold)
trace_mm_page_free_batched(page, cold);
free_hot_cold_page(page, cold);
}
+
+ INIT_LIST_HEAD(list);
+}
+
+/* Drop reference counts and free pages from a list */
+void free_pages_bulk(struct list_head *list)
+{
+ struct page *page, *next;
+ bool free_percpu = !in_interrupt();
+
+ list_for_each_entry_safe(page, next, list, lru) {
+ trace_mm_page_free_batched(page, 0);
+ if (put_page_testzero(page)) {
+ list_del(&page->lru);
+ if (free_percpu)
+ free_hot_cold_page(page, false);
+ else
+ __free_pages_ok(page, 0);
+ }
+ }
}
+EXPORT_SYMBOL_GPL(free_pages_bulk);
/*
* split_page takes a non-compound higher-order page, and splits it into
@@ -3887,6 +3908,99 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
EXPORT_SYMBOL(__alloc_pages_nodemask);
/*
+ * This is a batched version of the page allocator that attempts to
+ * allocate nr_pages quickly from the preferred zone and add them to list.
+ * Note that there is no guarantee that nr_pages will be allocated although
+ * every effort will be made to allocate at least one. Unlike the core
+ * allocator, no special effort is made to recover from transient
+ * failures caused by changes in cpusets. It should only be used from !IRQ
+ * context. An attempt to allocate a batch of patches from an interrupt
+ * will allocate a single page.
+ */
+unsigned long
+__alloc_pages_bulk_nodemask(gfp_t gfp_mask, unsigned int order,
+ struct zonelist *zonelist, nodemask_t *nodemask,
+ unsigned long nr_pages, struct list_head *alloc_list)
+{
+ struct page *page;
+ unsigned long alloced = 0;
+ unsigned int alloc_flags = ALLOC_WMARK_LOW;
+ struct zone *zone;
+ struct per_cpu_pages *pcp;
+ struct list_head *pcp_list;
+ int migratetype;
+ gfp_t alloc_mask = gfp_mask; /* The gfp_t that was actually used for allocation */
+ struct alloc_context ac = { };
+ bool cold = ((gfp_mask & __GFP_COLD) != 0);
+
+ /* If there are already pages on the list, don't bother */
+ if (!list_empty(alloc_list))
+ return 0;
+
+ /* Only handle bulk allocation of order-0 */
+ if (order || in_interrupt())
+ goto failed;
+
+ gfp_mask &= gfp_allowed_mask;
+ if (!prepare_alloc_pages(gfp_mask, order, zonelist, nodemask, &ac, &alloc_mask, &alloc_flags))
+ return 0;
+
+ finalise_ac(gfp_mask, order, &ac);
+ if (!ac.preferred_zoneref)
+ return 0;
+
+ /*
+ * Only attempt a batch allocation if watermarks on the preferred zone
+ * are safe.
+ */
+ zone = ac.preferred_zoneref->zone;
+ if (!zone_watermark_fast(zone, order, zone->watermark[ALLOC_WMARK_HIGH] + nr_pages,
+ zonelist_zone_idx(ac.preferred_zoneref), alloc_flags))
+ goto failed;
+
+ /* Attempt the batch allocation */
+ migratetype = ac.migratetype;
+
+ preempt_disable();
+ pcp = &this_cpu_ptr(zone->pageset)->pcp;
+ pcp_list = &pcp->lists[migratetype];
+
+ while (nr_pages) {
+ page = __rmqueue_pcplist(zone, order, gfp_mask, migratetype,
+ cold, pcp, pcp_list);
+ if (!page)
+ break;
+
+ prep_new_page(page, order, gfp_mask, 0);
+ nr_pages--;
+ alloced++;
+ list_add(&page->lru, alloc_list);
+ }
+
+ if (!alloced) {
+ preempt_enable_no_resched();
+ goto failed;
+ }
+
+ __count_zid_vm_events(PGALLOC, zone_idx(zone), alloced);
+ zone_statistics(zone, zone, gfp_mask);
+
+ preempt_enable_no_resched();
+
+ return alloced;
+
+failed:
+ page = __alloc_pages_nodemask(gfp_mask, order, zonelist, nodemask);
+ if (page) {
+ alloced++;
+ list_add(&page->lru, alloc_list);
+ }
+
+ return alloced;
+}
+EXPORT_SYMBOL(__alloc_pages_bulk_nodemask);
+
+/*
* Common helper functions.
*/
unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order)
--
2.11.0
[toc] | [next] | [standalone]
| From | "Hillf Danton" <hillf.zj@alibaba-inc.com> |
|---|---|
| Date | 2017-01-10 05:10 +0100 |
| Message-ID | <sXVUZ-4W8-5@gated-at.bofh.it> |
| In reply to | #1554464 |
On Tuesday, January 10, 2017 12:35 AM Mel Gorman wrote:
>
> This patch adds a new page allocator interface via alloc_pages_bulk,
> __alloc_pages_bulk and __alloc_pages_bulk_nodemask. A caller requests a
> number of pages to be allocated and added to a list. They can be freed in
> bulk using free_pages_bulk(). Note that it would theoretically be possible
> to use free_hot_cold_page_list for faster frees if the symbol was exported,
> the refcounts were 0 and the caller guaranteed it was not in an interrupt.
> This would be significantly faster in the free path but also more unsafer
> and a harder API to use.
>
> The API is not guaranteed to return the requested number of pages and
> may fail if the preferred allocation zone has limited free memory, the
> cpuset changes during the allocation or page debugging decides to fail
> an allocation. It's up to the caller to request more pages in batch if
> necessary.
>
> The following compares the allocation cost per page for different batch
> sizes. The baseline is allocating them one at a time and it compares with
> the performance when using the new allocation interface.
>
> pagealloc
> 4.10.0-rc2 4.10.0-rc2
> one-at-a-time bulk-v2
> Amean alloc-odr0-1 259.54 ( 0.00%) 106.62 ( 58.92%)
> Amean alloc-odr0-2 193.38 ( 0.00%) 76.38 ( 60.50%)
> Amean alloc-odr0-4 162.38 ( 0.00%) 57.23 ( 64.76%)
> Amean alloc-odr0-8 144.31 ( 0.00%) 48.77 ( 66.20%)
> Amean alloc-odr0-16 134.08 ( 0.00%) 45.38 ( 66.15%)
> Amean alloc-odr0-32 128.62 ( 0.00%) 42.77 ( 66.75%)
> Amean alloc-odr0-64 126.00 ( 0.00%) 41.00 ( 67.46%)
> Amean alloc-odr0-128 125.00 ( 0.00%) 40.08 ( 67.94%)
> Amean alloc-odr0-256 136.62 ( 0.00%) 56.00 ( 59.01%)
> Amean alloc-odr0-512 152.00 ( 0.00%) 69.00 ( 54.61%)
> Amean alloc-odr0-1024 158.00 ( 0.00%) 76.23 ( 51.75%)
> Amean alloc-odr0-2048 163.00 ( 0.00%) 81.15 ( 50.21%)
> Amean alloc-odr0-4096 169.77 ( 0.00%) 85.92 ( 49.39%)
> Amean alloc-odr0-8192 170.00 ( 0.00%) 88.00 ( 48.24%)
> Amean alloc-odr0-16384 170.00 ( 0.00%) 89.00 ( 47.65%)
> Amean free-odr0-1 88.69 ( 0.00%) 55.69 ( 37.21%)
> Amean free-odr0-2 66.00 ( 0.00%) 49.38 ( 25.17%)
> Amean free-odr0-4 54.23 ( 0.00%) 45.38 ( 16.31%)
> Amean free-odr0-8 48.23 ( 0.00%) 44.23 ( 8.29%)
> Amean free-odr0-16 47.00 ( 0.00%) 45.00 ( 4.26%)
> Amean free-odr0-32 44.77 ( 0.00%) 43.92 ( 1.89%)
> Amean free-odr0-64 44.00 ( 0.00%) 43.00 ( 2.27%)
> Amean free-odr0-128 43.00 ( 0.00%) 43.00 ( 0.00%)
> Amean free-odr0-256 60.69 ( 0.00%) 60.46 ( 0.38%)
> Amean free-odr0-512 79.23 ( 0.00%) 76.00 ( 4.08%)
> Amean free-odr0-1024 86.00 ( 0.00%) 85.38 ( 0.72%)
> Amean free-odr0-2048 91.00 ( 0.00%) 91.23 ( -0.25%)
> Amean free-odr0-4096 94.85 ( 0.00%) 95.62 ( -0.81%)
> Amean free-odr0-8192 97.00 ( 0.00%) 97.00 ( 0.00%)
> Amean free-odr0-16384 98.00 ( 0.00%) 97.46 ( 0.55%)
> Amean total-odr0-1 348.23 ( 0.00%) 162.31 ( 53.39%)
> Amean total-odr0-2 259.38 ( 0.00%) 125.77 ( 51.51%)
> Amean total-odr0-4 216.62 ( 0.00%) 102.62 ( 52.63%)
> Amean total-odr0-8 192.54 ( 0.00%) 93.00 ( 51.70%)
> Amean total-odr0-16 181.08 ( 0.00%) 90.38 ( 50.08%)
> Amean total-odr0-32 173.38 ( 0.00%) 86.69 ( 50.00%)
> Amean total-odr0-64 170.00 ( 0.00%) 84.00 ( 50.59%)
> Amean total-odr0-128 168.00 ( 0.00%) 83.08 ( 50.55%)
> Amean total-odr0-256 197.31 ( 0.00%) 116.46 ( 40.97%)
> Amean total-odr0-512 231.23 ( 0.00%) 145.00 ( 37.29%)
> Amean total-odr0-1024 244.00 ( 0.00%) 161.62 ( 33.76%)
> Amean total-odr0-2048 254.00 ( 0.00%) 172.38 ( 32.13%)
> Amean total-odr0-4096 264.62 ( 0.00%) 181.54 ( 31.40%)
> Amean total-odr0-8192 267.00 ( 0.00%) 185.00 ( 30.71%)
> Amean total-odr0-16384 268.00 ( 0.00%) 186.46 ( 30.42%)
>
> It shows a roughly 50-60% reduction in the cost of allocating pages.
> The free paths are not improved as much but relatively little can be batched
> there. It's not quite as fast as it could be but taking further shortcuts
> would require making a lot of assumptions about the state of the page and
> the context of the caller.
>
> Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
> ---
Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com>
> include/linux/gfp.h | 24 +++++++++++
> mm/page_alloc.c | 116 +++++++++++++++++++++++++++++++++++++++++++++++++++-
> 2 files changed, 139 insertions(+), 1 deletion(-)
>
> diff --git a/include/linux/gfp.h b/include/linux/gfp.h
> index 4175dca4ac39..b2fe171ee1c4 100644
> --- a/include/linux/gfp.h
> +++ b/include/linux/gfp.h
> @@ -433,6 +433,29 @@ __alloc_pages(gfp_t gfp_mask, unsigned int order,
> return __alloc_pages_nodemask(gfp_mask, order, zonelist, NULL);
> }
>
> +unsigned long
> +__alloc_pages_bulk_nodemask(gfp_t gfp_mask, unsigned int order,
> + struct zonelist *zonelist, nodemask_t *nodemask,
> + unsigned long nr_pages, struct list_head *alloc_list);
> +
> +static inline unsigned long
> +__alloc_pages_bulk(gfp_t gfp_mask, unsigned int order,
> + struct zonelist *zonelist, unsigned long nr_pages,
> + struct list_head *list)
> +{
> + return __alloc_pages_bulk_nodemask(gfp_mask, order, zonelist, NULL,
> + nr_pages, list);
> +}
> +
> +static inline unsigned long
> +alloc_pages_bulk(gfp_t gfp_mask, unsigned int order,
> + unsigned long nr_pages, struct list_head *list)
> +{
> + int nid = numa_mem_id();
> + return __alloc_pages_bulk(gfp_mask, order,
> + node_zonelist(nid, gfp_mask), nr_pages, list);
> +}
> +
> /*
> * Allocate pages, preferring the node given as nid. The node must be valid and
> * online. For more general interface, see alloc_pages_node().
> @@ -504,6 +527,7 @@ extern void __free_pages(struct page *page, unsigned int order);
> extern void free_pages(unsigned long addr, unsigned int order);
> extern void free_hot_cold_page(struct page *page, bool cold);
> extern void free_hot_cold_page_list(struct list_head *list, bool cold);
> +extern void free_pages_bulk(struct list_head *list);
>
> struct page_frag_cache;
> extern void __page_frag_drain(struct page *page, unsigned int order,
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 232cadbe9231..4f142270fbf0 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -2485,7 +2485,7 @@ void free_hot_cold_page(struct page *page, bool cold)
> }
>
> /*
> - * Free a list of 0-order pages
> + * Free a list of 0-order pages whose reference count is already zero.
> */
> void free_hot_cold_page_list(struct list_head *list, bool cold)
> {
> @@ -2495,7 +2495,28 @@ void free_hot_cold_page_list(struct list_head *list, bool cold)
> trace_mm_page_free_batched(page, cold);
> free_hot_cold_page(page, cold);
> }
> +
> + INIT_LIST_HEAD(list);
Nit: can we cut this overhead off?
> +}
> +
> +/* Drop reference counts and free pages from a list */
> +void free_pages_bulk(struct list_head *list)
> +{
> + struct page *page, *next;
> + bool free_percpu = !in_interrupt();
> +
> + list_for_each_entry_safe(page, next, list, lru) {
> + trace_mm_page_free_batched(page, 0);
> + if (put_page_testzero(page)) {
> + list_del(&page->lru);
> + if (free_percpu)
> + free_hot_cold_page(page, false);
> + else
> + __free_pages_ok(page, 0);
> + }
> + }
> }
> +EXPORT_SYMBOL_GPL(free_pages_bulk);
>
> /*
> * split_page takes a non-compound higher-order page, and splits it into
> @@ -3887,6 +3908,99 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
> EXPORT_SYMBOL(__alloc_pages_nodemask);
>
> /*
> + * This is a batched version of the page allocator that attempts to
> + * allocate nr_pages quickly from the preferred zone and add them to list.
> + * Note that there is no guarantee that nr_pages will be allocated although
> + * every effort will be made to allocate at least one. Unlike the core
> + * allocator, no special effort is made to recover from transient
> + * failures caused by changes in cpusets. It should only be used from !IRQ
> + * context. An attempt to allocate a batch of patches from an interrupt
> + * will allocate a single page.
> + */
> +unsigned long
> +__alloc_pages_bulk_nodemask(gfp_t gfp_mask, unsigned int order,
> + struct zonelist *zonelist, nodemask_t *nodemask,
> + unsigned long nr_pages, struct list_head *alloc_list)
> +{
> + struct page *page;
> + unsigned long alloced = 0;
> + unsigned int alloc_flags = ALLOC_WMARK_LOW;
> + struct zone *zone;
> + struct per_cpu_pages *pcp;
> + struct list_head *pcp_list;
> + int migratetype;
> + gfp_t alloc_mask = gfp_mask; /* The gfp_t that was actually used for allocation */
> + struct alloc_context ac = { };
> + bool cold = ((gfp_mask & __GFP_COLD) != 0);
> +
> + /* If there are already pages on the list, don't bother */
> + if (!list_empty(alloc_list))
> + return 0;
Nit: can we move the check to the call site?
> +
> + /* Only handle bulk allocation of order-0 */
> + if (order || in_interrupt())
> + goto failed;
Ditto
Hillf
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2017-01-10 09:40 +0100 |
| Message-ID | <sY08i-7ry-25@gated-at.bofh.it> |
| In reply to | #1554910 |
On Tue, Jan 10, 2017 at 12:00:27PM +0800, Hillf Danton wrote:
> > It shows a roughly 50-60% reduction in the cost of allocating pages.
> > The free paths are not improved as much but relatively little can be batched
> > there. It's not quite as fast as it could be but taking further shortcuts
> > would require making a lot of assumptions about the state of the page and
> > the context of the caller.
> >
> > Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
> > ---
> Acked-by: Hillf Danton <hillf.zj@alibaba-inc.com>
>
Thanks.
> > @@ -2485,7 +2485,7 @@ void free_hot_cold_page(struct page *page, bool cold)
> > }
> >
> > /*
> > - * Free a list of 0-order pages
> > + * Free a list of 0-order pages whose reference count is already zero.
> > */
> > void free_hot_cold_page_list(struct list_head *list, bool cold)
> > {
> > @@ -2495,7 +2495,28 @@ void free_hot_cold_page_list(struct list_head *list, bool cold)
> > trace_mm_page_free_batched(page, cold);
> > free_hot_cold_page(page, cold);
> > }
> > +
> > + INIT_LIST_HEAD(list);
>
> Nit: can we cut this overhead off?
Yes, but note that any caller of free_hot_cold_page_list() is then
required to reinit the list themselves or it'll cause corruption. It's
unlikely that a user of the bulk interface will handle the refcounts and
be able to use this interface properly but if they do, they need to
either reinit this or add the hunk back in.
It happens that all callers currently don't care.
> > /*
> > * split_page takes a non-compound higher-order page, and splits it into
> > @@ -3887,6 +3908,99 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
> > EXPORT_SYMBOL(__alloc_pages_nodemask);
> >
> > /*
> > + * This is a batched version of the page allocator that attempts to
> > + * allocate nr_pages quickly from the preferred zone and add them to list.
> > + * Note that there is no guarantee that nr_pages will be allocated although
> > + * every effort will be made to allocate at least one. Unlike the core
> > + * allocator, no special effort is made to recover from transient
> > + * failures caused by changes in cpusets. It should only be used from !IRQ
> > + * context. An attempt to allocate a batch of patches from an interrupt
> > + * will allocate a single page.
> > + */
> > +unsigned long
> > +__alloc_pages_bulk_nodemask(gfp_t gfp_mask, unsigned int order,
> > + struct zonelist *zonelist, nodemask_t *nodemask,
> > + unsigned long nr_pages, struct list_head *alloc_list)
> > +{
> > + struct page *page;
> > + unsigned long alloced = 0;
> > + unsigned int alloc_flags = ALLOC_WMARK_LOW;
> > + struct zone *zone;
> > + struct per_cpu_pages *pcp;
> > + struct list_head *pcp_list;
> > + int migratetype;
> > + gfp_t alloc_mask = gfp_mask; /* The gfp_t that was actually used for allocation */
> > + struct alloc_context ac = { };
> > + bool cold = ((gfp_mask & __GFP_COLD) != 0);
> > +
> > + /* If there are already pages on the list, don't bother */
> > + if (!list_empty(alloc_list))
> > + return 0;
>
> Nit: can we move the check to the call site?
Yes, but it makes the API slightly more hazardous to use.
> > +
> > + /* Only handle bulk allocation of order-0 */
> > + if (order || in_interrupt())
> > + goto failed;
>
> Ditto
>
Same, if the caller is in interrupt context, there is a slight risk that
they'll corrupt the list in a manner that will be tricky to catch. The
checks are to minimise the risk of being surprising.
--
Mel Gorman
SUSE Labs
[toc] | [prev] | [next] | [standalone]
| From | Jesper Dangaard Brouer <brouer@redhat.com> |
|---|---|
| Date | 2017-01-16 15:30 +0100 |
| Message-ID | <t0gsi-1yN-23@gated-at.bofh.it> |
| In reply to | #1554464 |
On Mon, 9 Jan 2017 16:35:18 +0000 Mel Gorman <mgorman@techsingularity.net> wrote: > This patch adds a new page allocator interface via alloc_pages_bulk, > __alloc_pages_bulk and __alloc_pages_bulk_nodemask. A caller requests a > number of pages to be allocated and added to a list. They can be freed in > bulk using free_pages_bulk(). Note that it would theoretically be possible > to use free_hot_cold_page_list for faster frees if the symbol was exported, > the refcounts were 0 and the caller guaranteed it was not in an interrupt. > This would be significantly faster in the free path but also more unsafer > and a harder API to use. > > The API is not guaranteed to return the requested number of pages and > may fail if the preferred allocation zone has limited free memory, the > cpuset changes during the allocation or page debugging decides to fail > an allocation. It's up to the caller to request more pages in batch if > necessary. > > The following compares the allocation cost per page for different batch > sizes. The baseline is allocating them one at a time and it compares with > the performance when using the new allocation interface. I've also played with testing the bulking API here: [1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/bench/page_bench04_bulk.c My baseline single (order-0 page) show: 158 cycles(tsc) 39.593 ns Using bulking API: Bulk: 1 cycles: 128 nanosec: 32.134 Bulk: 2 cycles: 107 nanosec: 26.783 Bulk: 3 cycles: 100 nanosec: 25.047 Bulk: 4 cycles: 95 nanosec: 23.988 Bulk: 8 cycles: 91 nanosec: 22.823 Bulk: 16 cycles: 88 nanosec: 22.093 Bulk: 32 cycles: 85 nanosec: 21.338 Bulk: 64 cycles: 85 nanosec: 21.315 Bulk: 128 cycles: 84 nanosec: 21.214 Bulk: 256 cycles: 115 nanosec: 28.979 This bulk API (and other improvements part of patchset) definitely moves the speed of the page allocator closer to my (crazy) time budget target of between 201 to 269 cycles per packet[1]. Remember I was reporting[2] order-0 cost between 231 to 277 cycles, at MM-summit 2016, so this is a huge improvement since then. The bulk numbers are great, but it still cannot compete with the recycles tricks used by drivers. Looking at the code (and as Mel also mentions) there is room for improvements especially on the bulk free-side. [1] http://people.netfilter.org/hawk/presentations/devconf2016/net_stack_challenges_100G_Feb2016.pdf [2] http://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf > pagealloc > 4.10.0-rc2 4.10.0-rc2 > one-at-a-time bulk-v2 > Amean alloc-odr0-1 259.54 ( 0.00%) 106.62 ( 58.92%) > Amean alloc-odr0-2 193.38 ( 0.00%) 76.38 ( 60.50%) > Amean alloc-odr0-4 162.38 ( 0.00%) 57.23 ( 64.76%) > Amean alloc-odr0-8 144.31 ( 0.00%) 48.77 ( 66.20%) > Amean alloc-odr0-16 134.08 ( 0.00%) 45.38 ( 66.15%) > Amean alloc-odr0-32 128.62 ( 0.00%) 42.77 ( 66.75%) > Amean alloc-odr0-64 126.00 ( 0.00%) 41.00 ( 67.46%) > Amean alloc-odr0-128 125.00 ( 0.00%) 40.08 ( 67.94%) > Amean alloc-odr0-256 136.62 ( 0.00%) 56.00 ( 59.01%) > Amean alloc-odr0-512 152.00 ( 0.00%) 69.00 ( 54.61%) > Amean alloc-odr0-1024 158.00 ( 0.00%) 76.23 ( 51.75%) > Amean alloc-odr0-2048 163.00 ( 0.00%) 81.15 ( 50.21%) > Amean alloc-odr0-4096 169.77 ( 0.00%) 85.92 ( 49.39%) > Amean alloc-odr0-8192 170.00 ( 0.00%) 88.00 ( 48.24%) > Amean alloc-odr0-16384 170.00 ( 0.00%) 89.00 ( 47.65%) > Amean free-odr0-1 88.69 ( 0.00%) 55.69 ( 37.21%) > Amean free-odr0-2 66.00 ( 0.00%) 49.38 ( 25.17%) > Amean free-odr0-4 54.23 ( 0.00%) 45.38 ( 16.31%) > Amean free-odr0-8 48.23 ( 0.00%) 44.23 ( 8.29%) > Amean free-odr0-16 47.00 ( 0.00%) 45.00 ( 4.26%) > Amean free-odr0-32 44.77 ( 0.00%) 43.92 ( 1.89%) > Amean free-odr0-64 44.00 ( 0.00%) 43.00 ( 2.27%) > Amean free-odr0-128 43.00 ( 0.00%) 43.00 ( 0.00%) > Amean free-odr0-256 60.69 ( 0.00%) 60.46 ( 0.38%) > Amean free-odr0-512 79.23 ( 0.00%) 76.00 ( 4.08%) > Amean free-odr0-1024 86.00 ( 0.00%) 85.38 ( 0.72%) > Amean free-odr0-2048 91.00 ( 0.00%) 91.23 ( -0.25%) > Amean free-odr0-4096 94.85 ( 0.00%) 95.62 ( -0.81%) > Amean free-odr0-8192 97.00 ( 0.00%) 97.00 ( 0.00%) > Amean free-odr0-16384 98.00 ( 0.00%) 97.46 ( 0.55%) > Amean total-odr0-1 348.23 ( 0.00%) 162.31 ( 53.39%) > Amean total-odr0-2 259.38 ( 0.00%) 125.77 ( 51.51%) > Amean total-odr0-4 216.62 ( 0.00%) 102.62 ( 52.63%) > Amean total-odr0-8 192.54 ( 0.00%) 93.00 ( 51.70%) > Amean total-odr0-16 181.08 ( 0.00%) 90.38 ( 50.08%) > Amean total-odr0-32 173.38 ( 0.00%) 86.69 ( 50.00%) > Amean total-odr0-64 170.00 ( 0.00%) 84.00 ( 50.59%) > Amean total-odr0-128 168.00 ( 0.00%) 83.08 ( 50.55%) > Amean total-odr0-256 197.31 ( 0.00%) 116.46 ( 40.97%) > Amean total-odr0-512 231.23 ( 0.00%) 145.00 ( 37.29%) > Amean total-odr0-1024 244.00 ( 0.00%) 161.62 ( 33.76%) > Amean total-odr0-2048 254.00 ( 0.00%) 172.38 ( 32.13%) > Amean total-odr0-4096 264.62 ( 0.00%) 181.54 ( 31.40%) > Amean total-odr0-8192 267.00 ( 0.00%) 185.00 ( 30.71%) > Amean total-odr0-16384 268.00 ( 0.00%) 186.46 ( 30.42%) > > It shows a roughly 50-60% reduction in the cost of allocating pages. > The free paths are not improved as much but relatively little can be batched > there. It's not quite as fast as it could be but taking further shortcuts > would require making a lot of assumptions about the state of the page and > the context of the caller. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer
[toc] | [prev] | [next] | [standalone]
| From | Mel Gorman <mgorman@techsingularity.net> |
|---|---|
| Date | 2017-01-16 16:10 +0100 |
| Message-ID | <t0h4Z-23f-9@gated-at.bofh.it> |
| In reply to | #1559785 |
On Mon, Jan 16, 2017 at 03:25:18PM +0100, Jesper Dangaard Brouer wrote: > On Mon, 9 Jan 2017 16:35:18 +0000 > Mel Gorman <mgorman@techsingularity.net> wrote: > > > This patch adds a new page allocator interface via alloc_pages_bulk, > > __alloc_pages_bulk and __alloc_pages_bulk_nodemask. A caller requests a > > number of pages to be allocated and added to a list. They can be freed in > > bulk using free_pages_bulk(). Note that it would theoretically be possible > > to use free_hot_cold_page_list for faster frees if the symbol was exported, > > the refcounts were 0 and the caller guaranteed it was not in an interrupt. > > This would be significantly faster in the free path but also more unsafer > > and a harder API to use. > > > > The API is not guaranteed to return the requested number of pages and > > may fail if the preferred allocation zone has limited free memory, the > > cpuset changes during the allocation or page debugging decides to fail > > an allocation. It's up to the caller to request more pages in batch if > > necessary. > > > > The following compares the allocation cost per page for different batch > > sizes. The baseline is allocating them one at a time and it compares with > > the performance when using the new allocation interface. > > I've also played with testing the bulking API here: > [1] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/bench/page_bench04_bulk.c > > My baseline single (order-0 page) show: 158 cycles(tsc) 39.593 ns > > Using bulking API: > Bulk: 1 cycles: 128 nanosec: 32.134 > Bulk: 2 cycles: 107 nanosec: 26.783 > Bulk: 3 cycles: 100 nanosec: 25.047 > Bulk: 4 cycles: 95 nanosec: 23.988 > Bulk: 8 cycles: 91 nanosec: 22.823 > Bulk: 16 cycles: 88 nanosec: 22.093 > Bulk: 32 cycles: 85 nanosec: 21.338 > Bulk: 64 cycles: 85 nanosec: 21.315 > Bulk: 128 cycles: 84 nanosec: 21.214 > Bulk: 256 cycles: 115 nanosec: 28.979 > > This bulk API (and other improvements part of patchset) definitely > moves the speed of the page allocator closer to my (crazy) time budget > target of between 201 to 269 cycles per packet[1]. Remember I was > reporting[2] order-0 cost between 231 to 277 cycles, at MM-summit > 2016, so this is a huge improvement since then. > Good to hear. > The bulk numbers are great, but it still cannot compete with the > recycles tricks used by drivers. Looking at the code (and as Mel also > mentions) there is room for improvements especially on the bulk free-side. > A major component there is how the ref handling is done and the safety checks. If necessary, you could mandate that callers drop the reference count or allow pages to be freed with an elevated count to avoid the atomic ops. In an early prototype, I made the refcount "mistake" and freeing was half the cost. I restored it in the final version to have an API that was almost identical to the existing allocator other than the bulking aspects. You could also disable all the other safety checks and flag that the bulk alloc/free potentially frees pages in inconsistent state. That would increase the performance at the cost of safety but that may be acceptable given that driver recycling of pages also avoids the same checks. You could also consider disabling the statistics updates to avoid a bunch of per-cpu stat operations, particularly if the pages were mostly recycled by the generic pool allocator. -- Mel Gorman SUSE Labs
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web