Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1677897

[PATCH v4 01/10] x86/mm: Don't reenter flush_tlb_func_common()

Show all headers | View raw

It was historically possible to have two concurrent TLB flushes
targetting the same CPU: one initiated locally and one initiated
remotely.  This can now cause an OOPS in leave_mm() at
arch/x86/mm/tlb.c:47:

        if (this_cpu_read(cpu_tlbstate.state) == TLBSTATE_OK)
                BUG();

with this call trace:
 flush_tlb_func_local arch/x86/mm/tlb.c:239 [inline]
 flush_tlb_mm_range+0x26d/0x370 arch/x86/mm/tlb.c:317

Without reentrancy, this OOPS is impossible: leave_mm() is only
called if we're not in TLBSTATE_OK, but then we're unexpectedly
in TLBSTATE_OK in leave_mm().

This can be caused by flush_tlb_func_remote() happening between
the two checks and calling leave_mm(), resulting in two consecutive
leave_mm() calls on the same CPU with no intervening switch_mm()
calls.

We never saw this OOPS before because the old leave_mm()
implementation didn't put us back in TLBSTATE_OK, so the assertion
didn't fire.

Nadav noticed the reentrancy issue in a different context, but
neither of us realized that it caused a problem yet.

Cc: Dave Hansen <dave.hansen@intel.com>
Reviewed-by: Nadav Amit <nadav.amit@gmail.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Reported-by: "Levin, Alexander (Sasha Levin)" <alexander.levin@verizon.com>
Fixes: 3d28ebceaffa ("x86/mm: Rework lazy TLB to track the actual loaded mm")
Signed-off-by: Andy Lutomirski <luto@kernel.org>
---
 arch/x86/mm/tlb.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c
index b2485d69f7c2..1cc47838d1e8 100644
--- a/arch/x86/mm/tlb.c
+++ b/arch/x86/mm/tlb.c
@@ -192,6 +192,9 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,
 static void flush_tlb_func_common(const struct flush_tlb_info *f,
 				  bool local, enum tlb_flush_reason reason)
 {
+	/* This code cannot presently handle being reentered. */
+	VM_WARN_ON(!irqs_disabled());
+
 	if (this_cpu_read(cpu_tlbstate.state) != TLBSTATE_OK) {
 		leave_mm(smp_processor_id());
 		return;
@@ -297,8 +300,13 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start,
 		info.end = TLB_FLUSH_ALL;
 	}
 
-	if (mm == this_cpu_read(cpu_tlbstate.loaded_mm))
+	if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) {
+		VM_WARN_ON(irqs_disabled());
+		local_irq_disable();
 		flush_tlb_func_local(&info, TLB_LOCAL_MM_SHOOTDOWN);
+		local_irq_enable();
+	}
+
 	if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids)
 		flush_tlb_others(mm_cpumask(mm), &info);
 	put_cpu();
@@ -354,8 +362,13 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 
 	int cpu = get_cpu();
 
-	if (cpumask_test_cpu(cpu, &batch->cpumask))
+	if (cpumask_test_cpu(cpu, &batch->cpumask)) {
+		VM_WARN_ON(irqs_disabled());
+		local_irq_disable();
 		flush_tlb_func_local(&info, TLB_LOCAL_SHOOTDOWN);
+		local_irq_enable();
+	}
+
 	if (cpumask_any_but(&batch->cpumask, cpu) < nr_cpu_ids)
 		flush_tlb_others(&batch->cpumask, &info);
 	cpumask_clear(&batch->cpumask);
-- 
2.9.4

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH v4 00/10] PCID and improved laziness Andy Lutomirski <luto@kernel.org> - 2017-06-29 18:00 +0200
  [PATCH v4 02/10] x86/mm: Delete a big outdated comment about TLB flushing Andy Lutomirski <luto@kernel.org> - 2017-06-29 18:00 +0200
    [tip:x86/mm] x86/mm: Delete a big outdated comment about TLB  flushing tip-bot for Andy Lutomirski <tipbot@zytor.com> - 2017-06-30 15:20 +0200
  [PATCH v4 09/10] x86/mm: Enable CR4.PCIDE on supported systems Andy Lutomirski <luto@kernel.org> - 2017-06-29 18:00 +0200
  [PATCH v4 08/10] x86/mm: Add nopcid to turn off PCID Andy Lutomirski <luto@kernel.org> - 2017-06-29 18:00 +0200
  [PATCH v4 05/10] x86/mm: Rework lazy TLB mode and TLB freshness tracking Andy Lutomirski <luto@kernel.org> - 2017-06-29 18:00 +0200
  [PATCH v4 03/10] x86/mm: Give each mm TLB flush generation a unique ID Andy Lutomirski <luto@kernel.org> - 2017-06-29 18:00 +0200
  [PATCH v4 06/10] x86/mm: Stop calling leave_mm() in idle code Andy Lutomirski <luto@kernel.org> - 2017-06-29 18:00 +0200
  [PATCH v4 01/10] x86/mm: Don't reenter flush_tlb_func_common() Andy Lutomirski <luto@kernel.org> - 2017-06-29 18:00 +0200
    [tip:x86/mm] x86/mm: Don't reenter flush_tlb_func_common() tip-bot for Andy Lutomirski <tipbot@zytor.com> - 2017-06-30 15:20 +0200
  [PATCH v4 04/10] x86/mm: Track the TLB's tlb_gen and update the flushing algorithm Andy Lutomirski <luto@kernel.org> - 2017-06-29 18:00 +0200
  Re: [PATCH v4 00/10] PCID and improved laziness Matt Fleming <matt@codeblueprint.co.uk> - 2017-06-30 14:50 +0200

csiph-web