Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1601158 > unrolled thread

[PATCH v2 0/5] mm: support parallel free of memory

Back to article view | Back to linux.kernel

Contents

  [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-15 10:10 +0100
    [PATCH v2 2/5] mm: parallel free pages Aaron Lu <aaron.lu@intel.com> - 2017-03-15 10:10 +0100
      Re: [PATCH v2 2/5] mm: parallel free pages "Hillf Danton" <hillf.zj@alibaba-inc.com> - 2017-03-15 10:50 +0100
        Re: [PATCH v2 2/5] mm: parallel free pages Aaron Lu <aaron.lu@intel.com> - 2017-03-15 13:00 +0100
    Re: [PATCH v2 0/5] mm: support parallel free of memory Michal Hocko <mhocko@kernel.org> - 2017-03-15 15:30 +0100
      Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-15 16:50 +0100
        Re: [PATCH v2 0/5] mm: support parallel free of memory Michal Hocko <mhocko@kernel.org> - 2017-03-15 17:30 +0100
          Re: [PATCH v2 0/5] mm: support parallel free of memory Tim Chen <tim.c.chen@linux.intel.com> - 2017-03-15 22:40 +0100
            Re: [PATCH v2 0/5] mm: support parallel free of memory Michal Hocko <mhocko@kernel.org> - 2017-03-16 10:10 +0100
              Re: [PATCH v2 0/5] mm: support parallel free of memory Tim Chen <tim.c.chen@linux.intel.com> - 2017-03-16 19:40 +0100
                Re: [PATCH v2 0/5] mm: support parallel free of memory Minchan Kim <minchan@kernel.org> - 2017-03-17 09:10 +0100
                Re: [PATCH v2 0/5] mm: support parallel free of memory Michal Hocko <mhocko@kernel.org> - 2017-03-17 09:30 +0100
                  Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-17 13:40 +0100
                    Re: [PATCH v2 0/5] mm: support parallel free of memory Michal Hocko <mhocko@kernel.org> - 2017-03-17 14:00 +0100
                    Re: [PATCH v2 0/5] mm: support parallel free of memory Peter Zijlstra <peterz@infradead.org> - 2017-03-17 14:20 +0100
                  Re: [PATCH v2 0/5] mm: support parallel free of memory Peter Zijlstra <peterz@infradead.org> - 2017-03-17 14:00 +0100
                    Re: [PATCH v2 0/5] mm: support parallel free of memory Michal Hocko <mhocko@kernel.org> - 2017-03-17 14:30 +0100
              Re: [PATCH v2 0/5] mm: support parallel free of memory Dave Hansen <dave.hansen@intel.com> - 2017-03-21 16:10 +0100
                Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-22 09:20 +0100
                Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-24 08:10 +0100
              Re: [PATCH v2 0/5] mm: support parallel free of memory Tim Chen <tim.c.chen@linux.intel.com> - 2017-03-21 16:20 +0100
          Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-16 08:00 +0100
          Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-16 08:40 +0100
            Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-16 15:00 +0100
              Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-16 15:20 +0100
    Re: [PATCH v2 0/5] mm: support parallel free of memory Vlastimil Babka <vbabka@suse.cz> - 2017-03-15 16:00 +0100
      Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-15 17:00 +0100
      Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-17 04:20 +0100
    Re: [PATCH v2 0/5] mm: support parallel free of memory Aaron Lu <aaron.lu@intel.com> - 2017-03-17 03:30 +0100

Page 2 of 2 — ← Prev page 1 [2]

#1605705

On Thu, 2017-03-16 at 10:07 +0100, Michal Hocko wrote:
> 
> > > the main problem is that kworkers will not belong to the same cpu group
> > > and so they will not be throttled properly.
> > You do have a point that this page freeing activities should strive to
> > affect other threads not in the same cgroup minimally.
> > 
> > On the other hand, we also don't do this throttling of kworkers 
> > today (e.g. pdflush) according to the cgroup it is doing work for.
> Yes, I am not saying this a new problem. I just wanted to point out that
> this is something to consider here. I believe this should be fixable.
> Worker can attach to the same cgroup the initiator had for example
> (assuming the cgroup core allows that which is something would have to
> be checked).

Instead of attaching the kworders to the cgroup of the initiator, I
wonder what people think about creating a separate kworker cgroup. 
The administrator can set limit on its cpu resource bandwidth
if he/she does not want such kworkers perturbing the system.

Tim

[toc] | [prev] | [next] | [standalone]

#1602004

On Wed, Mar 15, 2017 at 05:28:43PM +0100, Michal Hocko wrote:
> On Wed 15-03-17 23:44:07, Aaron Lu wrote:
> > On Wed, Mar 15, 2017 at 03:18:14PM +0100, Michal Hocko wrote:
> > > On Wed 15-03-17 16:59:59, Aaron Lu wrote:
> > > [...]
> > > > The proposed parallel free did this: if the process has many pages to be
> > > > freed, accumulate them in these struct mmu_gather_batch(es) one after
> > > > another till 256K pages are accumulated. Then take this singly linked
> > > > list starting from tlb->local.next off struct mmu_gather *tlb and free
> > > > them in a worker thread. The main thread can return to continue zap
> > > > other pages(after freeing pages pointed by tlb->local.pages).
> > > 
> > > I didn't have a look at the implementation yet but there are two
> > > concerns that raise up from this description. Firstly how are we going
> > > to tune the number of workers. I assume there will be some upper bound
> > > (one of the patch subject mentions debugfs for tuning) and secondly
> > 
> > The workers are put in a dedicated workqueue which is introduced in
> > patch 3/5 and the number of workers can be tuned through that workqueue's
> > sysfs interface: max_active.
> 
> I suspect we cannot expect users to tune this. What do you consider a
> reasonable default?

I agree with Tim that 4 is a reasonable number for now.

> 
> Moreover, and this is a more generic question, is this functionality
> useful in general purpose workloads? After all the amount of the work to

I'm not sure. The main motivation is to speed up the exit of the crashed
application as explained by Dave.

> be done is the same we just risk more lock contentions, unexpected CPU
> usage etc. Which workloads will benefit from having exit path faster?
>  
> > > if we offload the page freeing to the worker then the original context
> > > can consume much more cpu cycles than it was configured via cpu
> 
> I was not precise here. I meant to say more cpu cycles per time unit
> that it was allowed.
> 
> > > controller. How are we going to handle that? Or is this considered
> > > acceptable?
> > 
> > I'll need to think about and take a look at this subject(not familiar
> > with cpu controller).
> 
> the main problem is that kworkers will not belong to the same cpu group
> and so they will not be throttled properly.

Looks like a fundamental problem as long as kworker is used.
With the default max_active of the workqueue set to 4, do you think this
is a blocking issue?

Thanks.

[toc] | [prev] | [next] | [standalone]

#1602022

On Wed, Mar 15, 2017 at 05:28:43PM +0100, Michal Hocko wrote:
... ...
> After all the amount of the work to be done is the same we just risk
> more lock contentions, unexpected CPU usage etc.

I start to realize this is a good question.

I guess max_active=4 produced almost the best result(max_active=8 is
only slightly better) is due to the test box is a 4 node machine and
therefore, there are 4 zone->lock to contend(let's ignore those tiny
zones only available in node 0).

I'm going to test on a EP to see if max_active=2 will suffice to produce
a good enough result. If so, the proper default number should be the
number of nodes.

Thanks.

[toc] | [prev] | [next] | [standalone]

#1602297

On Thu, Mar 16, 2017 at 03:34:03PM +0800, Aaron Lu wrote:
> On Wed, Mar 15, 2017 at 05:28:43PM +0100, Michal Hocko wrote:
> ... ...
> > After all the amount of the work to be done is the same we just risk
> > more lock contentions, unexpected CPU usage etc.
> 
> I start to realize this is a good question.
> 
> I guess max_active=4 produced almost the best result(max_active=8 is
> only slightly better) is due to the test box is a 4 node machine and
> therefore, there are 4 zone->lock to contend(let's ignore those tiny
> zones only available in node 0).
> 
> I'm going to test on a EP to see if max_active=2 will suffice to produce
> a good enough result. If so, the proper default number should be the
> number of nodes.

Here are test results on 2 nodes EP with 128GiB memory, test size 100GiB.

max_active           time
vanilla              2.971s ±3.8%
2                    1.699s ±13.7%
4                    1.616s ±3.1%
8                    1.642s ±0.9%

So 4 gives best result but 2 is probably good enough.

If the size each worker deals with is changed from 1G to 2G:

max_active           time
2                    1.605s ±1.7%
4                    1.639s ±1.2%
8                    1.626s ±1.8%

Considering that we are mostly improving for memory intensive apps, the
default setting should probably be: max_active = node_number with each
worker freeing 2G memory.

[toc] | [prev] | [next] | [standalone]

#1602318

On Thu, Mar 16, 2017 at 09:51:22PM +0800, Aaron Lu wrote:
> Considering that we are mostly improving for memory intensive apps, the
> default setting should probably be: max_active = node_number with each
> worker freeing 2G memory.

In case people want to give this setting a try, here is what to do.

On 2-nodes EP:
# echo 2 > /sys/devices/virtual/workqueue/batch_free_wq/max_active
# echo 1030 > /sys/kernel/debug/parallel_free/max_gather_batch_count

On 4-nodes EX:
# echo 4 > /sys/devices/virtual/workqueue/batch_free_wq/max_active
# echo 1030 > /sys/kernel/debug/parallel_free/max_gather_batch_count

[toc] | [prev] | [next] | [standalone]

#1601467

On 03/15/2017 09:59 AM, Aaron Lu wrote:
> For regular processes, the time taken in its exit() path to free its
> used memory is not a problem. But there are heavy ones that consume
> several Terabytes memory and the time taken to free its memory in its
> exit() path could last more than ten minutes if THP is not used.
> 
> As Dave Hansen explained why do this in kernel:
> "
> One of the places we saw this happen was when an app crashed and was
> exit()'ing under duress without cleaning up nicely.  The time that it
> takes to unmap a few TB of 4k pages is pretty excessive.
> "

Yeah, it would be nice to improve such cases.

> To optimize this use case, a parallel free method is proposed here and
> it is based on the current gather batch free(the following description
> is taken from patch 2/5's changelog).
> 
> The current gather batch free works like this:
> For each struct mmu_gather *tlb, there is a static buffer to store those
> to-be-freed page pointers. The size is MMU_GATHER_BUNDLE, which is
> defined to be 8. So if a tlb tear down doesn't free more than 8 pages,
> that is all we need. If 8+ pages are to be freed, new pages will need
> to be allocated to store those to-be-freed page pointers.
> 
> The structure used to describe the saved page pointers is called
> struct mmu_gather_batch and tlb->local is of this type. tlb->local is
> different than other struct mmu_gather_batch(es) in that the page
> pointer array used by tlb->local points to the previouslly described
> static buffer while the other struct mmu_gather_batch(es) page pointer
> array points to the dynamically allocated pages.
> 
> These batches will form a singly linked list, starting from &tlb->local.
> 
> tlb->local.pages  => tlb->pages(8 pointers)
>       \|/
>       next => batch1->pages => about 510 pointers
>                 \|/
>                 next => batch2->pages => about 510 pointers
>                           \|/
>                           next => batch3->pages => about 510 pointers
>                                     ... ...
> 
> The proposed parallel free did this: if the process has many pages to be
> freed, accumulate them in these struct mmu_gather_batch(es) one after
> another till 256K pages are accumulated. Then take this singly linked
> list starting from tlb->local.next off struct mmu_gather *tlb and free
> them in a worker thread. The main thread can return to continue zap
> other pages(after freeing pages pointed by tlb->local.pages).
> 
> A test program that did a single malloc() of 320G memory is used to see
> how useful the proposed parallel free solution is, the time calculated
> is for the free() call. Test machine is a Haswell EX which has
> 4nodes/72cores/144threads with 512G memory. All tests are done with THP
> disabled.
> 
> kernel                             time
> v4.10                              10.8s  ±2.8%
> this patch(with default setting)   5.795s ±5.8%

I wonder if the difference would be larger if the parallelism was done
on a higher level, something around unmap_page_range(). IIUC the current
approach still leaves a lot of work to a single thread, right?
I assume it would be more complicated, but doable as we already have the
OOM reaper doing unmaps parallel to other activity? Has that been
considered?

Thanks, Vlastimil

> 
> Patch 3/5 introduced a dedicated workqueue for the free workers and
> here are more results when setting different values for max_active of
> this workqueue:
> 
> max_active:   time
> 1             8.9s   ±0.5%
> 2             5.65s  ±5.5%
> 4             4.84s  ±0.16%
> 8             4.77s  ±0.97%
> 16            4.85s  ±0.77%
> 32            6.21s  ±0.46%
> 
> Comments are welcome and appreciated.
> 
> v2 changes: Nothing major, only minor ones.
>  - rebased on top of v4.11-rc2-mmotm-2017-03-14-15-41;
>  - use list_add_tail instead of list_add to add worker to tlb's worker
>    list so that when doing flush, the first queued worker gets flushed
>    first(based on the comsumption that the first queued worker has a
>    better chance of finishing its job than those later queued workers);
>  - use bool instead of int for variable free_batch_page in function
>    tlb_flush_mmu_free_batches;
>  - style change according to ./scripts/checkpatch;
>  - reword some of the changelogs to make it more readable.
> 
> v1 is here:
> https://lkml.org/lkml/2017/2/24/245
> 
> Aaron Lu (5):
>   mm: add tlb_flush_mmu_free_batches
>   mm: parallel free pages
>   mm: use a dedicated workqueue for the free workers
>   mm: add force_free_pages in zap_pte_range
>   mm: add debugfs interface for parallel free tuning
> 
>  include/asm-generic/tlb.h |  15 ++---
>  mm/memory.c               | 141 +++++++++++++++++++++++++++++++++++++++-------
>  2 files changed, 128 insertions(+), 28 deletions(-)
> 

[toc] | [prev] | [next] | [standalone]

#1601507

On Wed, Mar 15, 2017 at 03:56:02PM +0100, Vlastimil Babka wrote:
> On 03/15/2017 09:59 AM, Aaron Lu wrote:
> > For regular processes, the time taken in its exit() path to free its
> > used memory is not a problem. But there are heavy ones that consume
> > several Terabytes memory and the time taken to free its memory in its
> > exit() path could last more than ten minutes if THP is not used.
> > 
> > As Dave Hansen explained why do this in kernel:
> > "
> > One of the places we saw this happen was when an app crashed and was
> > exit()'ing under duress without cleaning up nicely.  The time that it
> > takes to unmap a few TB of 4k pages is pretty excessive.
> > "
> 
> Yeah, it would be nice to improve such cases.

Glad to hear this.

> 
> > To optimize this use case, a parallel free method is proposed here and
> > it is based on the current gather batch free(the following description
> > is taken from patch 2/5's changelog).
> > 
> > The current gather batch free works like this:
> > For each struct mmu_gather *tlb, there is a static buffer to store those
> > to-be-freed page pointers. The size is MMU_GATHER_BUNDLE, which is
> > defined to be 8. So if a tlb tear down doesn't free more than 8 pages,
> > that is all we need. If 8+ pages are to be freed, new pages will need
> > to be allocated to store those to-be-freed page pointers.
> > 
> > The structure used to describe the saved page pointers is called
> > struct mmu_gather_batch and tlb->local is of this type. tlb->local is
> > different than other struct mmu_gather_batch(es) in that the page
> > pointer array used by tlb->local points to the previouslly described
> > static buffer while the other struct mmu_gather_batch(es) page pointer
> > array points to the dynamically allocated pages.
> > 
> > These batches will form a singly linked list, starting from &tlb->local.
> > 
> > tlb->local.pages  => tlb->pages(8 pointers)
> >       \|/
> >       next => batch1->pages => about 510 pointers
> >                 \|/
> >                 next => batch2->pages => about 510 pointers
> >                           \|/
> >                           next => batch3->pages => about 510 pointers
> >                                     ... ...
> > 
> > The proposed parallel free did this: if the process has many pages to be
> > freed, accumulate them in these struct mmu_gather_batch(es) one after
> > another till 256K pages are accumulated. Then take this singly linked
> > list starting from tlb->local.next off struct mmu_gather *tlb and free
> > them in a worker thread. The main thread can return to continue zap
> > other pages(after freeing pages pointed by tlb->local.pages).
> > 
> > A test program that did a single malloc() of 320G memory is used to see
> > how useful the proposed parallel free solution is, the time calculated
> > is for the free() call. Test machine is a Haswell EX which has
> > 4nodes/72cores/144threads with 512G memory. All tests are done with THP
> > disabled.
> > 
> > kernel                             time
> > v4.10                              10.8s  ±2.8%
> > this patch(with default setting)   5.795s ±5.8%
> 
> I wonder if the difference would be larger if the parallelism was done
> on a higher level, something around unmap_page_range(). IIUC the current

We have tried to do it at the VMA level, but there is a problem: suppose
a program has many VMAs but only one or two of them are big/huge ones,
the parallism is not good.

I also considered PUD based parallel free, the potential issue with it
is: there could be very few physical pages actually present for that
PUD so the worker may have very few things to do.
For the test case used here though, PUD based one should work better
since all PTEs are faulted in.

> approach still leaves a lot of work to a single thread, right?

Yes, the main thread will be responsible for page table walk, PTE clear
and possibly flushing TLB in race condition. But considering the issues
of the other two mentioned approaches, I chose the current approach.

Perhaps I should also implement a PUD based parallel free and then use a
program that has 2 huge VMAs with equal size, one with all pages faulted
in RAM while the other has none and then compare the two approaches'
performance, does this make sense?

> I assume it would be more complicated, but doable as we already have the
> OOM reaper doing unmaps parallel to other activity? Has that been
> considered?

Since the tlb structure is not meant to be accessed concurrently, I
assume there will be some trouble to handle it if going the PUD based
approach. Will take a look at it tomorrow(it's late here).

Thanks.
-Aaron

> 
> Thanks, Vlastimil
> 
> > 
> > Patch 3/5 introduced a dedicated workqueue for the free workers and
> > here are more results when setting different values for max_active of
> > this workqueue:
> > 
> > max_active:   time
> > 1             8.9s   ±0.5%
> > 2             5.65s  ±5.5%
> > 4             4.84s  ±0.16%
> > 8             4.77s  ±0.97%
> > 16            4.85s  ±0.77%
> > 32            6.21s  ±0.46%
> > 
> > Comments are welcome and appreciated.
> > 
> > v2 changes: Nothing major, only minor ones.
> >  - rebased on top of v4.11-rc2-mmotm-2017-03-14-15-41;
> >  - use list_add_tail instead of list_add to add worker to tlb's worker
> >    list so that when doing flush, the first queued worker gets flushed
> >    first(based on the comsumption that the first queued worker has a
> >    better chance of finishing its job than those later queued workers);
> >  - use bool instead of int for variable free_batch_page in function
> >    tlb_flush_mmu_free_batches;
> >  - style change according to ./scripts/checkpatch;
> >  - reword some of the changelogs to make it more readable.
> > 
> > v1 is here:
> > https://lkml.org/lkml/2017/2/24/245
> > 
> > Aaron Lu (5):
> >   mm: add tlb_flush_mmu_free_batches
> >   mm: parallel free pages
> >   mm: use a dedicated workqueue for the free workers
> >   mm: add force_free_pages in zap_pte_range
> >   mm: add debugfs interface for parallel free tuning
> > 
> >  include/asm-generic/tlb.h |  15 ++---
> >  mm/memory.c               | 141 +++++++++++++++++++++++++++++++++++++++-------
> >  2 files changed, 128 insertions(+), 28 deletions(-)
> > 
> 

[toc] | [prev] | [next] | [standalone]

#1603007

On Wed, Mar 15, 2017 at 03:56:02PM +0100, Vlastimil Babka wrote:
> I wonder if the difference would be larger if the parallelism was done
> on a higher level, something around unmap_page_range(). IIUC the current

I guess I misunderstand you in my last email - doing it at
unmap_page_range() level is essentially doing it at a per-VMA level
since it is the main function used in unmap_single_vma(). We have tried
that and felt that it's not flexible as the proposed approach since
it wouldn't parallize well for:
1 work load that uses only 1 or very few huge VMA;
2 work load that has a lot of small VMAs.

The code is nice and easy though(developed at v4.9 time frame):

From f6d5cfde888b9e0356719fabe8754fdfe6fe236b Mon Sep 17 00:00:00 2001
From: Aaron Lu <aaron.lu@intel.com>
Date: Wed, 11 Jan 2017 15:56:06 +0800
Subject: [PATCH] mm: async free vma

---
 include/linux/mm_types.h |  6 ++++++
 mm/memory.c              | 23 ++++++++++++++++++++++-
 2 files changed, 28 insertions(+), 1 deletion(-)

diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 4a8acedf4b7d..d10d2ce8f8f4 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -358,6 +358,12 @@ struct vm_area_struct {
 	struct mempolicy *vm_policy;	/* NUMA policy for the VMA */
 #endif
 	struct vm_userfaultfd_ctx vm_userfaultfd_ctx;
+
+	struct vma_free_ctx {
+		unsigned long start_addr;
+		unsigned long end_addr;
+		struct work_struct work;
+	} free_ctx;
 };
 
 struct core_thread {
diff --git a/mm/memory.c b/mm/memory.c
index e18c57bdc75c..0fe4e45a044b 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -1345,6 +1345,17 @@ static void unmap_single_vma(struct mmu_gather *tlb,
 	}
 }
 
+static void unmap_single_vma_work(struct work_struct *work)
+{
+	struct vma_free_ctx *ctx = container_of(work, struct vma_free_ctx, work);
+	struct vm_area_struct *vma = container_of(ctx, struct vm_area_struct, free_ctx);
+	struct mmu_gather tlb;
+
+	tlb_gather_mmu(&tlb, vma->vm_mm, ctx->start_addr, ctx->end_addr);
+	unmap_single_vma(&tlb, vma, ctx->start_addr, ctx->end_addr, NULL);
+	tlb_finish_mmu(&tlb, ctx->start_addr, ctx->end_addr);
+}
+
 /**
  * unmap_vmas - unmap a range of memory covered by a list of vma's
  * @tlb: address of the caller's struct mmu_gather
@@ -1368,10 +1379,20 @@ void unmap_vmas(struct mmu_gather *tlb,
 		unsigned long end_addr)
 {
 	struct mm_struct *mm = vma->vm_mm;
+	struct vma_free_ctx *ctx;
+	struct vm_area_struct *tmp = vma;
 
 	mmu_notifier_invalidate_range_start(mm, start_addr, end_addr);
+	for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next) {
+		ctx = &vma->free_ctx;
+		ctx->start_addr = start_addr;
+		ctx->end_addr = end_addr;
+		INIT_WORK(&ctx->work, unmap_single_vma_work);
+		queue_work(system_unbound_wq, &ctx->work);
+	}
+	vma = tmp;
 	for ( ; vma && vma->vm_start < end_addr; vma = vma->vm_next)
-		unmap_single_vma(tlb, vma, start_addr, end_addr, NULL);
+		flush_work(&vma->free_ctx.work);
 	mmu_notifier_invalidate_range_end(mm, start_addr, end_addr);
 }
 
-- 
2.9.3

[toc] | [prev] | [next] | [standalone]

#1602995

On Thu, Mar 16, 2017 at 02:38:44PM -0500, Alex Thorlton wrote:
> On Wed, Mar 15, 2017 at 04:59:59PM +0800, Aaron Lu wrote:
> > v2 changes: Nothing major, only minor ones.
> >  - rebased on top of v4.11-rc2-mmotm-2017-03-14-15-41;
> >  - use list_add_tail instead of list_add to add worker to tlb's worker
> >    list so that when doing flush, the first queued worker gets flushed
> >    first(based on the comsumption that the first queued worker has a
> >    better chance of finishing its job than those later queued workers);
> >  - use bool instead of int for variable free_batch_page in function
> >    tlb_flush_mmu_free_batches;
> >  - style change according to ./scripts/checkpatch;
> >  - reword some of the changelogs to make it more readable.
> > 
> > v1 is here:
> > https://lkml.org/lkml/2017/2/24/245
> 
> I tested v1 on a Haswell system with 64 sockets/1024 cores/2048 threads
> and 8TB of RAM, with a 1TB malloc.  The average free() time for a 1TB
> malloc on a vanilla kernel was 41.69s, the patched kernel averaged
> 21.56s for the same test.

Thanks a lot for the test result.

> 
> I am testing v2 now and will report back with results in the next day or
> so.

Testing plain v2 shouldn't bring any surprise/difference, better set the
following param before the test(I'm planning to make them default in the
next version):
# echo 64 > /sys/devices/virtual/workqueue/batch_free_wq/max_active
# echo 1030 > /sys/kernel/debug/parallel_free/max_gather_batch_count

Regards,
Aaron

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web