Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1701793 > unrolled thread

[PATCH v6 6/7] mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  [PATCH v6 6/7] mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem Nadav Amit <namit@vmware.com> - 2017-08-02 09:40 +0200
    Re: [lkp-robot] [mm]  7674270022:  will-it-scale.per_process_ops  -19.3% regression Minchan Kim <minchan@kernel.org> - 2017-08-08 04:30 +0200
      Re: [lkp-robot] [mm]  7674270022:  will-it-scale.per_process_ops  -19.3% regression Nadav Amit <nadav.amit@gmail.com> - 2017-08-08 06:30 +0200
        Re: [lkp-robot] [mm]  7674270022:  will-it-scale.per_process_ops  -19.3% regression Nadav Amit <nadav.amit@gmail.com> - 2017-08-08 08:00 +0200
          Re: [lkp-robot] [mm]  7674270022:  will-it-scale.per_process_ops  -19.3% regression Minchan Kim <minchan@kernel.org> - 2017-08-08 10:10 +0200
            Re: [lkp-robot] [mm]  7674270022:  will-it-scale.per_process_ops  -19.3% regression Ye Xiaolong <xiaolong.ye@intel.com> - 2017-08-09 03:30 +0200
            Re: [lkp-robot] [mm]  7674270022:  will-it-scale.per_process_ops  -19.3% regression Ye Xiaolong <xiaolong.ye@intel.com> - 2017-08-09 05:10 +0200
              Re: [lkp-robot] [mm]  7674270022:  will-it-scale.per_process_ops  -19.3% regression Minchan Kim <minchan@kernel.org> - 2017-08-10 06:20 +0200
                Re: [lkp-robot] [mm]  7674270022:  will-it-scale.per_process_ops  -19.3% regression Nadav Amit <nadav.amit@gmail.com> - 2017-08-10 06:20 +0200
                  Re: [lkp-robot] [mm]  7674270022:  will-it-scale.per_process_ops  -19.3% regression Minchan Kim <minchan@kernel.org> - 2017-08-10 06:30 +0200
    Re: [PATCH v6 6/7] mm: fix MADV_[FREE|DONTNEED] TLB flush miss  problem Peter Zijlstra <peterz@infradead.org> - 2017-08-11 15:40 +0200

#1701793 — [PATCH v6 6/7] mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem

From: Minchan Kim <minchan@kernel.org>

Nadav reported parallel MADV_DONTNEED on same range has a stale TLB problem
and Mel fixed it[1] and found same problem on MADV_FREE[2].

Quote from Mel Gorman

"The race in question is CPU 0 running madv_free and updating some PTEs
while CPU 1 is also running madv_free and looking at the same PTEs.  CPU 1
may have writable TLB entries for a page but fail the pte_dirty check
(because CPU 0 has updated it already) and potentially fail to flush.
Hence, when madv_free on CPU 1 returns, there are still potentially
writable TLB entries and the underlying PTE is still present so that a
subsequent write does not necessarily propagate the dirty bit to the
underlying PTE any more.  Reclaim at some unknown time at the future may
then see that the PTE is still clean and discard the page even though a
write has happened in the meantime.  I think this is possible but I could
have missed some protection in madv_free that prevents it happening."

This patch aims for solving both problems all at once and is ready for
other problem with KSM, MADV_FREE and soft-dirty story[3].

TLB batch API(tlb_[gather|finish]_mmu] uses [inc|dec]_tlb_flush_pending and
mmu_tlb_flush_pending so that when tlb_finish_mmu is called, we can catch
there are parallel threads going on. In that case, forcefully, flush TLB to
prevent for user to access memory via stale TLB entry although it fail to
gather page table entry.

I confirmed this patch works with [4] test program Nadav gave so this patch
supersedes "mm: Always flush VMA ranges affected by zap_page_range v2" in
current mmotm.

NOTE:

This patch modifies arch-specific TLB gathering interface(x86, ia64, s390,
sh, um). It seems most of architecture are straightforward but s390 need to
be careful because tlb_flush_mmu works only if mm->context.flush_mm is set
to non-zero which happens only a pte entry really is cleared by
ptep_get_and_clear and friends. However, this problem never changes the pte
entries but need to flush to prevent memory access from stale tlb.

[1] http://lkml.kernel.org/r/20170725101230.5v7gvnjmcnkzzql3@techsingularity.net
[2] http://lkml.kernel.org/r/20170725100722.2dxnmgypmwnrfawp@suse.de
[3] http://lkml.kernel.org/r/BD3A0EBE-ECF4-41D4-87FA-C755EA9AB6BD@gmail.com
[4] https://patchwork.kernel.org/patch/9861621/

Cc: Ingo Molnar <mingo@redhat.com>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: linux-arch@vger.kernel.org
Reported-by: Nadav Amit <namit@vmware.com>
Reported-by: Mel Gorman <mgorman@techsingularity.net>
Signed-off-by: Minchan Kim <minchan@kernel.org>
Signed-off-by: Nadav Amit <namit@vmware.com>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
---
 arch/arm/include/asm/tlb.h  |  7 ++++++-
 arch/ia64/include/asm/tlb.h |  4 +++-
 arch/s390/include/asm/tlb.h |  7 ++++++-
 arch/sh/include/asm/tlb.h   |  4 ++--
 arch/um/include/asm/tlb.h   |  7 ++++++-
 include/asm-generic/tlb.h   |  2 +-
 include/linux/mm_types.h    |  8 ++++++++
 mm/memory.c                 | 17 +++++++++++++++--
 8 files changed, 47 insertions(+), 9 deletions(-)

diff --git a/arch/arm/include/asm/tlb.h b/arch/arm/include/asm/tlb.h
index 7f5b2a2d3861..d5562f9ce600 100644
--- a/arch/arm/include/asm/tlb.h
+++ b/arch/arm/include/asm/tlb.h
@@ -168,8 +168,13 @@ arch_tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm,
 
 static inline void
 arch_tlb_finish_mmu(struct mmu_gather *tlb,
-			unsigned long start, unsigned long end)
+			unsigned long start, unsigned long end, bool force)
 {
+	if (force) {
+		tlb->range_start = start;
+		tlb->range_end = end;
+	}
+
 	tlb_flush_mmu(tlb);
 
 	/* keep the page table cache within bounds */
diff --git a/arch/ia64/include/asm/tlb.h b/arch/ia64/include/asm/tlb.h
index 93cadc04ac62..cbe5ac3699bf 100644
--- a/arch/ia64/include/asm/tlb.h
+++ b/arch/ia64/include/asm/tlb.h
@@ -187,8 +187,10 @@ arch_tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm,
  */
 static inline void
 arch_tlb_finish_mmu(struct mmu_gather *tlb,
-			unsigned long start, unsigned long end)
+			unsigned long start, unsigned long end, bool force)
 {
+	if (force)
+		tlb->need_flush = 1;
 	/*
 	 * Note: tlb->nr may be 0 at this point, so we can't rely on tlb->start_addr and
 	 * tlb->end_addr.
diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h
index 0e59ef57e234..b20a3621024f 100644
--- a/arch/s390/include/asm/tlb.h
+++ b/arch/s390/include/asm/tlb.h
@@ -77,8 +77,13 @@ static inline void tlb_flush_mmu(struct mmu_gather *tlb)
 
 static inline void
 arch_tlb_finish_mmu(struct mmu_gather *tlb,
-		unsigned long start, unsigned long end)
+		unsigned long start, unsigned long end, bool force)
 {
+	if (force) {
+		tlb->start = start;
+		tlb->end = end;
+	}
+
 	tlb_flush_mmu(tlb);
 }
 
diff --git a/arch/sh/include/asm/tlb.h b/arch/sh/include/asm/tlb.h
index 89786560dbd4..51a8bc967e75 100644
--- a/arch/sh/include/asm/tlb.h
+++ b/arch/sh/include/asm/tlb.h
@@ -49,9 +49,9 @@ arch_tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm,
 
 static inline void
 arch_tlb_finish_mmu(struct mmu_gather *tlb,
-		unsigned long start, unsigned long end)
+		unsigned long start, unsigned long end, bool force)
 {
-	if (tlb->fullmm)
+	if (tlb->fullmm || force)
 		flush_tlb_mm(tlb->mm);
 
 	/* keep the page table cache within bounds */
diff --git a/arch/um/include/asm/tlb.h b/arch/um/include/asm/tlb.h
index 2a901eca7145..344d95619d03 100644
--- a/arch/um/include/asm/tlb.h
+++ b/arch/um/include/asm/tlb.h
@@ -87,8 +87,13 @@ tlb_flush_mmu(struct mmu_gather *tlb)
  */
 static inline void
 arch_tlb_finish_mmu(struct mmu_gather *tlb,
-		unsigned long start, unsigned long end)
+		unsigned long start, unsigned long end, bool force)
 {
+	if (force) {
+		tlb->start = start;
+		tlb->end = end;
+		tlb->need_flush = 1;
+	}
 	tlb_flush_mmu(tlb);
 
 	/* keep the page table cache within bounds */
diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h
index 8f71521e7a44..faddde44de8c 100644
--- a/include/asm-generic/tlb.h
+++ b/include/asm-generic/tlb.h
@@ -116,7 +116,7 @@ void arch_tlb_gather_mmu(struct mmu_gather *tlb,
 	struct mm_struct *mm, unsigned long start, unsigned long end);
 void tlb_flush_mmu(struct mmu_gather *tlb);
 void arch_tlb_finish_mmu(struct mmu_gather *tlb,
-			 unsigned long start, unsigned long end);
+			 unsigned long start, unsigned long end, bool force);
 extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page,
 				   int page_size);
 
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index fc44315df47a..664c1e553228 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -534,6 +534,14 @@ static inline bool mm_tlb_flush_pending(struct mm_struct *mm)
 	return atomic_read(&mm->tlb_flush_pending) > 0;
 }
 
+/*
+ * Returns true if there are two above TLB batching threads in parallel.
+ */
+static inline bool mm_tlb_flush_nested(struct mm_struct *mm)
+{
+	return atomic_read(&mm->tlb_flush_pending) > 1;
+}
+
 static inline void init_tlb_flush_pending(struct mm_struct *mm)
 {
 	atomic_set(&mm->tlb_flush_pending, 0);
diff --git a/mm/memory.c b/mm/memory.c
index 7848b5030be0..d7a620dd183a 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -272,10 +272,13 @@ void tlb_flush_mmu(struct mmu_gather *tlb)
  *	that were required.
  */
 void arch_tlb_finish_mmu(struct mmu_gather *tlb,
-		unsigned long start, unsigned long end)
+		unsigned long start, unsigned long end, bool force)
 {
 	struct mmu_gather_batch *batch, *next;
 
+	if (force)
+		__tlb_adjust_range(tlb, start, end - start);
+
 	tlb_flush_mmu(tlb);
 
 	/* keep the page table cache within bounds */
@@ -404,12 +407,22 @@ void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm,
 			unsigned long start, unsigned long end)
 {
 	arch_tlb_gather_mmu(tlb, mm, start, end);
+	inc_tlb_flush_pending(tlb->mm);
 }
 
 void tlb_finish_mmu(struct mmu_gather *tlb,
 		unsigned long start, unsigned long end)
 {
-	arch_tlb_finish_mmu(tlb, start, end);
+	/*
+	 * If there are parallel threads are doing PTE changes on same range
+	 * under non-exclusive lock(e.g., mmap_sem read-side) but defer TLB
+	 * flush by batching, a thread has stable TLB entry can fail to flush
+	 * the TLB by observing pte_none|!pte_dirty, for example so flush TLB
+	 * forcefully if we detect parallel PTE batching threads.
+	 */
+	bool force = mm_tlb_flush_nested(tlb->mm);
+
+	arch_tlb_finish_mmu(tlb, start, end, force);
 }
 
 /*
-- 
2.11.0

[toc] | [next] | [standalone]

#1705968 — Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression

Hi,

On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot wrote:
> 
> Greeting,
> 
> FYI, we noticed a -19.3% regression of will-it-scale.per_process_ops due to commit:
> 
> 
> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem")
> url: https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate-prevent-racy-access-to-tlb_flush_pending/20170802-205715
> 
> 
> in testcase: will-it-scale
> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
> with following parameters:
> 
> 	nr_task: 16
> 	mode: process
> 	test: brk1
> 	cpufreq_governor: performance
> 
> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
> test-url: https://github.com/antonblanchard/will-it-scale

Thanks for the report.
Could you explain what kinds of workload you are testing?

Does it calls frequently madvise(MADV_DONTNEED) in parallel on multiple
threads?

[toc] | [prev] | [next] | [standalone]

#1706006 — Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression

Minchan Kim <minchan@kernel.org> wrote:

> Hi,
> 
> On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot wrote:
>> Greeting,
>> 
>> FYI, we noticed a -19.3% regression of will-it-scale.per_process_ops due to commit:
>> 
>> 
>> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem")
>> url: https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate-prevent-racy-access-to-tlb_flush_pending/20170802-205715
>> 
>> 
>> in testcase: will-it-scale
>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
>> with following parameters:
>> 
>> 	nr_task: 16
>> 	mode: process
>> 	test: brk1
>> 	cpufreq_governor: performance
>> 
>> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
>> test-url: https://github.com/antonblanchard/will-it-scale
> 
> Thanks for the report.
> Could you explain what kinds of workload you are testing?
> 
> Does it calls frequently madvise(MADV_DONTNEED) in parallel on multiple
> threads?

According to the description it is "testcase:brk increase/decrease of one
page”. According to the mode it spawns multiple processes, not threads.

Since a single page is unmapped each time, and the iTLB-loads increase
dramatically, I would suspect that for some reason a full TLB flush is
caused during do_munmap().

If I find some free time, I’ll try to profile the workload - but feel free
to beat me to it.

Nadav 

[toc] | [prev] | [next] | [standalone]

#1706044 — Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression

Nadav Amit <nadav.amit@gmail.com> wrote:

> Minchan Kim <minchan@kernel.org> wrote:
> 
>> Hi,
>> 
>> On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot wrote:
>>> Greeting,
>>> 
>>> FYI, we noticed a -19.3% regression of will-it-scale.per_process_ops due to commit:
>>> 
>>> 
>>> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem")
>>> url: https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate-prevent-racy-access-to-tlb_flush_pending/20170802-205715
>>> 
>>> 
>>> in testcase: will-it-scale
>>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
>>> with following parameters:
>>> 
>>> 	nr_task: 16
>>> 	mode: process
>>> 	test: brk1
>>> 	cpufreq_governor: performance
>>> 
>>> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
>>> test-url: https://github.com/antonblanchard/will-it-scale
>> 
>> Thanks for the report.
>> Could you explain what kinds of workload you are testing?
>> 
>> Does it calls frequently madvise(MADV_DONTNEED) in parallel on multiple
>> threads?
> 
> According to the description it is "testcase:brk increase/decrease of one
> page”. According to the mode it spawns multiple processes, not threads.
> 
> Since a single page is unmapped each time, and the iTLB-loads increase
> dramatically, I would suspect that for some reason a full TLB flush is
> caused during do_munmap().
> 
> If I find some free time, I’ll try to profile the workload - but feel free
> to beat me to it.

The root-cause appears to be that tlb_finish_mmu() does not call
dec_tlb_flush_pending() - as it should. Any chance you can take care of it?

Having said that it appears that cpumask_any_but() is really inefficient
since it does not have an optimization for the case in which
small_const_nbits(nbits)==true. When I find some free time, I’ll try to deal
with it.

Thanks,
Nadav

[toc] | [prev] | [next] | [standalone]

#1706127 — Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression

On Mon, Aug 07, 2017 at 10:51:00PM -0700, Nadav Amit wrote:
> Nadav Amit <nadav.amit@gmail.com> wrote:
> 
> > Minchan Kim <minchan@kernel.org> wrote:
> > 
> >> Hi,
> >> 
> >> On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot wrote:
> >>> Greeting,
> >>> 
> >>> FYI, we noticed a -19.3% regression of will-it-scale.per_process_ops due to commit:
> >>> 
> >>> 
> >>> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem")
> >>> url: https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate-prevent-racy-access-to-tlb_flush_pending/20170802-205715
> >>> 
> >>> 
> >>> in testcase: will-it-scale
> >>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
> >>> with following parameters:
> >>> 
> >>> 	nr_task: 16
> >>> 	mode: process
> >>> 	test: brk1
> >>> 	cpufreq_governor: performance
> >>> 
> >>> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
> >>> test-url: https://github.com/antonblanchard/will-it-scale
> >> 
> >> Thanks for the report.
> >> Could you explain what kinds of workload you are testing?
> >> 
> >> Does it calls frequently madvise(MADV_DONTNEED) in parallel on multiple
> >> threads?
> > 
> > According to the description it is "testcase:brk increase/decrease of one
> > page”. According to the mode it spawns multiple processes, not threads.
> > 
> > Since a single page is unmapped each time, and the iTLB-loads increase
> > dramatically, I would suspect that for some reason a full TLB flush is
> > caused during do_munmap().
> > 
> > If I find some free time, I’ll try to profile the workload - but feel free
> > to beat me to it.
> 
> The root-cause appears to be that tlb_finish_mmu() does not call
> dec_tlb_flush_pending() - as it should. Any chance you can take care of it?

Oops, but with second looking, it seems it's not my fault. ;-)
https://marc.info/?l=linux-mm&m=150156699114088&w=2

Anyway, thanks for the pointing out.
xiaolong.ye, could you retest with this fix?

From 83012114c9cd9304f0d55d899bb4b9329d0e22ac Mon Sep 17 00:00:00 2001
From: Minchan Kim <minchan@kernel.org>
Date: Tue, 8 Aug 2017 17:05:19 +0900
Subject: [PATCH] mm: decrease tlb flush pending count in tlb_finish_mmu

The tlb pending count increased by tlb_gather_mmu should be decreased
at tlb_finish_mmu. Otherwise, A lot of TLB happens which makes
performance regression.

Signed-off-by: Minchan Kim <minchan@kernel.org>
---
 mm/memory.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/memory.c b/mm/memory.c
index 34b1fcb829e4..ad2617552f55 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -423,6 +423,7 @@ void tlb_finish_mmu(struct mmu_gather *tlb,
 	bool force = mm_tlb_flush_nested(tlb->mm);
 
 	arch_tlb_finish_mmu(tlb, start, end, force);
+	dec_tlb_flush_pending(tlb->mm);
 }
 
 /*
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1706993 — Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression

On 08/08, Minchan Kim wrote:
>On Mon, Aug 07, 2017 at 10:51:00PM -0700, Nadav Amit wrote:
>> Nadav Amit <nadav.amit@gmail.com> wrote:
>> 
>> > Minchan Kim <minchan@kernel.org> wrote:
>> > 
>> >> Hi,
>> >> 
>> >> On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot wrote:
>> >>> Greeting,
>> >>> 
>> >>> FYI, we noticed a -19.3% regression of will-it-scale.per_process_ops due to commit:
>> >>> 
>> >>> 
>> >>> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem")
>> >>> url: https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate-prevent-racy-access-to-tlb_flush_pending/20170802-205715
>> >>> 
>> >>> 
>> >>> in testcase: will-it-scale
>> >>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
>> >>> with following parameters:
>> >>> 
>> >>> 	nr_task: 16
>> >>> 	mode: process
>> >>> 	test: brk1
>> >>> 	cpufreq_governor: performance
>> >>> 
>> >>> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
>> >>> test-url: https://github.com/antonblanchard/will-it-scale
>> >> 
>> >> Thanks for the report.
>> >> Could you explain what kinds of workload you are testing?
>> >> 
>> >> Does it calls frequently madvise(MADV_DONTNEED) in parallel on multiple
>> >> threads?
>> > 
>> > According to the description it is "testcase:brk increase/decrease of one
>> > page”. According to the mode it spawns multiple processes, not threads.
>> > 
>> > Since a single page is unmapped each time, and the iTLB-loads increase
>> > dramatically, I would suspect that for some reason a full TLB flush is
>> > caused during do_munmap().
>> > 
>> > If I find some free time, I’ll try to profile the workload - but feel free
>> > to beat me to it.
>> 
>> The root-cause appears to be that tlb_finish_mmu() does not call
>> dec_tlb_flush_pending() - as it should. Any chance you can take care of it?
>
>Oops, but with second looking, it seems it's not my fault. ;-)
>https://marc.info/?l=linux-mm&m=150156699114088&w=2
>
>Anyway, thanks for the pointing out.
>xiaolong.ye, could you retest with this fix?

Sure, I'll provide the result later.

Thanks,
Xiaolong
>
>From 83012114c9cd9304f0d55d899bb4b9329d0e22ac Mon Sep 17 00:00:00 2001
>From: Minchan Kim <minchan@kernel.org>
>Date: Tue, 8 Aug 2017 17:05:19 +0900
>Subject: [PATCH] mm: decrease tlb flush pending count in tlb_finish_mmu
>
>The tlb pending count increased by tlb_gather_mmu should be decreased
>at tlb_finish_mmu. Otherwise, A lot of TLB happens which makes
>performance regression.
>
>Signed-off-by: Minchan Kim <minchan@kernel.org>
>---
> mm/memory.c | 1 +
> 1 file changed, 1 insertion(+)
>
>diff --git a/mm/memory.c b/mm/memory.c
>index 34b1fcb829e4..ad2617552f55 100644
>--- a/mm/memory.c
>+++ b/mm/memory.c
>@@ -423,6 +423,7 @@ void tlb_finish_mmu(struct mmu_gather *tlb,
> 	bool force = mm_tlb_flush_nested(tlb->mm);
> 
> 	arch_tlb_finish_mmu(tlb, start, end, force);
>+	dec_tlb_flush_pending(tlb->mm);
> }
> 
> /*
>-- 
>2.7.4
>

[toc] | [prev] | [next] | [standalone]

#1707029 — Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression

On 08/08, Minchan Kim wrote:
>On Mon, Aug 07, 2017 at 10:51:00PM -0700, Nadav Amit wrote:
>> Nadav Amit <nadav.amit@gmail.com> wrote:
>> 
>> > Minchan Kim <minchan@kernel.org> wrote:
>> > 
>> >> Hi,
>> >> 
>> >> On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot wrote:
>> >>> Greeting,
>> >>> 
>> >>> FYI, we noticed a -19.3% regression of will-it-scale.per_process_ops due to commit:
>> >>> 
>> >>> 
>> >>> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem")
>> >>> url: https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate-prevent-racy-access-to-tlb_flush_pending/20170802-205715
>> >>> 
>> >>> 
>> >>> in testcase: will-it-scale
>> >>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
>> >>> with following parameters:
>> >>> 
>> >>> 	nr_task: 16
>> >>> 	mode: process
>> >>> 	test: brk1
>> >>> 	cpufreq_governor: performance
>> >>> 
>> >>> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
>> >>> test-url: https://github.com/antonblanchard/will-it-scale
>> >> 
>> >> Thanks for the report.
>> >> Could you explain what kinds of workload you are testing?
>> >> 
>> >> Does it calls frequently madvise(MADV_DONTNEED) in parallel on multiple
>> >> threads?
>> > 
>> > According to the description it is "testcase:brk increase/decrease of one
>> > page”. According to the mode it spawns multiple processes, not threads.
>> > 
>> > Since a single page is unmapped each time, and the iTLB-loads increase
>> > dramatically, I would suspect that for some reason a full TLB flush is
>> > caused during do_munmap().
>> > 
>> > If I find some free time, I’ll try to profile the workload - but feel free
>> > to beat me to it.
>> 
>> The root-cause appears to be that tlb_finish_mmu() does not call
>> dec_tlb_flush_pending() - as it should. Any chance you can take care of it?
>
>Oops, but with second looking, it seems it's not my fault. ;-)
>https://marc.info/?l=linux-mm&m=150156699114088&w=2
>
>Anyway, thanks for the pointing out.
>xiaolong.ye, could you retest with this fix?
>

I've queued tests for 5 times and results show this patch (e8f682574e4 "mm:
decrease tlb flush pending count in tlb_finish_mmu") does help recover the
performance back.

378005bdbac0a2ec  76742700225cad9df49f053993  e8f682574e45b6406dadfffeb4  
----------------  --------------------------  --------------------------  
         %stddev      change         %stddev      change         %stddev
             \          |                \          |                \  
   3405093             -19%    2747088              -2%    3348752        will-it-scale.per_process_ops
      1280 ±  3%        -2%       1257 ±  3%        -6%       1207        vmstat.system.cs
      2702 ± 18%        11%       3002 ± 19%        17%       3156 ± 18%  numa-vmstat.node0.nr_mapped
     10765 ± 18%        11%      11964 ± 19%        17%      12588 ± 18%  numa-meminfo.node0.Mapped
      0.00 ± 47%       -40%       0.00 ± 45%       -84%       0.00 ± 42%  mpstat.cpu.soft%

Thanks,
Xiaolong


>From 83012114c9cd9304f0d55d899bb4b9329d0e22ac Mon Sep 17 00:00:00 2001
>From: Minchan Kim <minchan@kernel.org>
>Date: Tue, 8 Aug 2017 17:05:19 +0900
>Subject: [PATCH] mm: decrease tlb flush pending count in tlb_finish_mmu
>
>The tlb pending count increased by tlb_gather_mmu should be decreased
>at tlb_finish_mmu. Otherwise, A lot of TLB happens which makes
>performance regression.
>
>Signed-off-by: Minchan Kim <minchan@kernel.org>
>---
> mm/memory.c | 1 +
> 1 file changed, 1 insertion(+)
>
>diff --git a/mm/memory.c b/mm/memory.c
>index 34b1fcb829e4..ad2617552f55 100644
>--- a/mm/memory.c
>+++ b/mm/memory.c
>@@ -423,6 +423,7 @@ void tlb_finish_mmu(struct mmu_gather *tlb,
> 	bool force = mm_tlb_flush_nested(tlb->mm);
> 
> 	arch_tlb_finish_mmu(tlb, start, end, force);
>+	dec_tlb_flush_pending(tlb->mm);
> }
> 
> /*
>-- 
>2.7.4
>

[toc] | [prev] | [next] | [standalone]

#1708199 — Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression

On Wed, Aug 09, 2017 at 10:59:02AM +0800, Ye Xiaolong wrote:
> On 08/08, Minchan Kim wrote:
> >On Mon, Aug 07, 2017 at 10:51:00PM -0700, Nadav Amit wrote:
> >> Nadav Amit <nadav.amit@gmail.com> wrote:
> >> 
> >> > Minchan Kim <minchan@kernel.org> wrote:
> >> > 
> >> >> Hi,
> >> >> 
> >> >> On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot wrote:
> >> >>> Greeting,
> >> >>> 
> >> >>> FYI, we noticed a -19.3% regression of will-it-scale.per_process_ops due to commit:
> >> >>> 
> >> >>> 
> >> >>> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem")
> >> >>> url: https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate-prevent-racy-access-to-tlb_flush_pending/20170802-205715
> >> >>> 
> >> >>> 
> >> >>> in testcase: will-it-scale
> >> >>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
> >> >>> with following parameters:
> >> >>> 
> >> >>> 	nr_task: 16
> >> >>> 	mode: process
> >> >>> 	test: brk1
> >> >>> 	cpufreq_governor: performance
> >> >>> 
> >> >>> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
> >> >>> test-url: https://github.com/antonblanchard/will-it-scale
> >> >> 
> >> >> Thanks for the report.
> >> >> Could you explain what kinds of workload you are testing?
> >> >> 
> >> >> Does it calls frequently madvise(MADV_DONTNEED) in parallel on multiple
> >> >> threads?
> >> > 
> >> > According to the description it is "testcase:brk increase/decrease of one
> >> > page”. According to the mode it spawns multiple processes, not threads.
> >> > 
> >> > Since a single page is unmapped each time, and the iTLB-loads increase
> >> > dramatically, I would suspect that for some reason a full TLB flush is
> >> > caused during do_munmap().
> >> > 
> >> > If I find some free time, I’ll try to profile the workload - but feel free
> >> > to beat me to it.
> >> 
> >> The root-cause appears to be that tlb_finish_mmu() does not call
> >> dec_tlb_flush_pending() - as it should. Any chance you can take care of it?
> >
> >Oops, but with second looking, it seems it's not my fault. ;-)
> >https://marc.info/?l=linux-mm&m=150156699114088&w=2
> >
> >Anyway, thanks for the pointing out.
> >xiaolong.ye, could you retest with this fix?
> >
> 
> I've queued tests for 5 times and results show this patch (e8f682574e4 "mm:
> decrease tlb flush pending count in tlb_finish_mmu") does help recover the
> performance back.
> 
> 378005bdbac0a2ec  76742700225cad9df49f053993  e8f682574e45b6406dadfffeb4  
> ----------------  --------------------------  --------------------------  
>          %stddev      change         %stddev      change         %stddev
>              \          |                \          |                \  
>    3405093             -19%    2747088              -2%    3348752        will-it-scale.per_process_ops
>       1280 ±  3%        -2%       1257 ±  3%        -6%       1207        vmstat.system.cs
>       2702 ± 18%        11%       3002 ± 19%        17%       3156 ± 18%  numa-vmstat.node0.nr_mapped
>      10765 ± 18%        11%      11964 ± 19%        17%      12588 ± 18%  numa-meminfo.node0.Mapped
>       0.00 ± 47%       -40%       0.00 ± 45%       -84%       0.00 ± 42%  mpstat.cpu.soft%
> 
> Thanks,
> Xiaolong

Thanks for the testing!

[toc] | [prev] | [next] | [standalone]

#1708200 — Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression

Minchan Kim <minchan@kernel.org> wrote:

> On Wed, Aug 09, 2017 at 10:59:02AM +0800, Ye Xiaolong wrote:
>> On 08/08, Minchan Kim wrote:
>>> On Mon, Aug 07, 2017 at 10:51:00PM -0700, Nadav Amit wrote:
>>>> Nadav Amit <nadav.amit@gmail.com> wrote:
>>>> 
>>>>> Minchan Kim <minchan@kernel.org> wrote:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> On Tue, Aug 08, 2017 at 09:19:23AM +0800, kernel test robot wrote:
>>>>>>> Greeting,
>>>>>>> 
>>>>>>> FYI, we noticed a -19.3% regression of will-it-scale.per_process_ops due to commit:
>>>>>>> 
>>>>>>> 
>>>>>>> commit: 76742700225cad9df49f05399381ac3f1ec3dc60 ("mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem")
>>>>>>> url: https://github.com/0day-ci/linux/commits/Nadav-Amit/mm-migrate-prevent-racy-access-to-tlb_flush_pending/20170802-205715
>>>>>>> 
>>>>>>> 
>>>>>>> in testcase: will-it-scale
>>>>>>> on test machine: 88 threads Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz with 64G memory
>>>>>>> with following parameters:
>>>>>>> 
>>>>>>> 	nr_task: 16
>>>>>>> 	mode: process
>>>>>>> 	test: brk1
>>>>>>> 	cpufreq_governor: performance
>>>>>>> 
>>>>>>> test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
>>>>>>> test-url: https://github.com/antonblanchard/will-it-scale
>>>>>> 
>>>>>> Thanks for the report.
>>>>>> Could you explain what kinds of workload you are testing?
>>>>>> 
>>>>>> Does it calls frequently madvise(MADV_DONTNEED) in parallel on multiple
>>>>>> threads?
>>>>> 
>>>>> According to the description it is "testcase:brk increase/decrease of one
>>>>> page”. According to the mode it spawns multiple processes, not threads.
>>>>> 
>>>>> Since a single page is unmapped each time, and the iTLB-loads increase
>>>>> dramatically, I would suspect that for some reason a full TLB flush is
>>>>> caused during do_munmap().
>>>>> 
>>>>> If I find some free time, I’ll try to profile the workload - but feel free
>>>>> to beat me to it.
>>>> 
>>>> The root-cause appears to be that tlb_finish_mmu() does not call
>>>> dec_tlb_flush_pending() - as it should. Any chance you can take care of it?
>>> 
>>> Oops, but with second looking, it seems it's not my fault. ;-)
>>> https://marc.info/?l=linux-mm&m=150156699114088&w=2
>>> 
>>> Anyway, thanks for the pointing out.
>>> xiaolong.ye, could you retest with this fix?
>> 
>> I've queued tests for 5 times and results show this patch (e8f682574e4 "mm:
>> decrease tlb flush pending count in tlb_finish_mmu") does help recover the
>> performance back.
>> 
>> 378005bdbac0a2ec  76742700225cad9df49f053993  e8f682574e45b6406dadfffeb4  
>> ----------------  --------------------------  --------------------------  
>>         %stddev      change         %stddev      change         %stddev
>>             \          |                \          |                \  
>>   3405093             -19%    2747088              -2%    3348752        will-it-scale.per_process_ops
>>      1280 ±  3%        -2%       1257 ±  3%        -6%       1207        vmstat.system.cs
>>      2702 ± 18%        11%       3002 ± 19%        17%       3156 ± 18%  numa-vmstat.node0.nr_mapped
>>     10765 ± 18%        11%      11964 ± 19%        17%      12588 ± 18%  numa-meminfo.node0.Mapped
>>      0.00 ± 47%       -40%       0.00 ± 45%       -84%       0.00 ± 42%  mpstat.cpu.soft%
>> 
>> Thanks,
>> Xiaolong
> 
> Thanks for the testing!

Sorry again for screwing your patch, Minchan.

[toc] | [prev] | [next] | [standalone]

#1708202 — Re: [lkp-robot] [mm] 7674270022: will-it-scale.per_process_ops -19.3% regression

On Wed, Aug 09, 2017 at 09:14:50PM -0700, Nadav Amit wrote:

Hi Nadav,

< snip >

> >>>>> According to the description it is "testcase:brk increase/decrease of one
> >>>>> page”. According to the mode it spawns multiple processes, not threads.
> >>>>> 
> >>>>> Since a single page is unmapped each time, and the iTLB-loads increase
> >>>>> dramatically, I would suspect that for some reason a full TLB flush is
> >>>>> caused during do_munmap().
> >>>>> 
> >>>>> If I find some free time, I’ll try to profile the workload - but feel free
> >>>>> to beat me to it.
> >>>> 
> >>>> The root-cause appears to be that tlb_finish_mmu() does not call
> >>>> dec_tlb_flush_pending() - as it should. Any chance you can take care of it?
> >>> 
> >>> Oops, but with second looking, it seems it's not my fault. ;-)
> >>> https://marc.info/?l=linux-mm&m=150156699114088&w=2
> >>> 
> >>> Anyway, thanks for the pointing out.
> >>> xiaolong.ye, could you retest with this fix?
> >> 
> >> I've queued tests for 5 times and results show this patch (e8f682574e4 "mm:
> >> decrease tlb flush pending count in tlb_finish_mmu") does help recover the
> >> performance back.
> >> 
> >> 378005bdbac0a2ec  76742700225cad9df49f053993  e8f682574e45b6406dadfffeb4  
> >> ----------------  --------------------------  --------------------------  
> >>         %stddev      change         %stddev      change         %stddev
> >>             \          |                \          |                \  
> >>   3405093             -19%    2747088              -2%    3348752        will-it-scale.per_process_ops
> >>      1280 ±  3%        -2%       1257 ±  3%        -6%       1207        vmstat.system.cs
> >>      2702 ± 18%        11%       3002 ± 19%        17%       3156 ± 18%  numa-vmstat.node0.nr_mapped
> >>     10765 ± 18%        11%      11964 ± 19%        17%      12588 ± 18%  numa-meminfo.node0.Mapped
> >>      0.00 ± 47%       -40%       0.00 ± 45%       -84%       0.00 ± 42%  mpstat.cpu.soft%
> >> 
> >> Thanks,
> >> Xiaolong
> > 
> > Thanks for the testing!
> 
> Sorry again for screwing your patch, Minchan.

Never mind! It always happens. :)
In this chance, I really appreciates your insight/testing/cooperation!

[toc] | [prev] | [next] | [standalone]

#1709618 — Re: [PATCH v6 6/7] mm: fix MADV_[FREE|DONTNEED] TLB flush miss problem

On Tue, Aug 01, 2017 at 05:08:17PM -0700, Nadav Amit wrote:
>  void tlb_finish_mmu(struct mmu_gather *tlb,
>  		unsigned long start, unsigned long end)
>  {
> -	arch_tlb_finish_mmu(tlb, start, end);
> +	/*
> +	 * If there are parallel threads are doing PTE changes on same range
> +	 * under non-exclusive lock(e.g., mmap_sem read-side) but defer TLB
> +	 * flush by batching, a thread has stable TLB entry can fail to flush
> +	 * the TLB by observing pte_none|!pte_dirty, for example so flush TLB
> +	 * forcefully if we detect parallel PTE batching threads.
> +	 */
> +	bool force = mm_tlb_flush_nested(tlb->mm);
> +
> +	arch_tlb_finish_mmu(tlb, start, end, force);
>  }

I don't understand the comment nor the ordering. What guarantees we see
the increment if we need to?

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web