Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1535433
| From | Borislav Petkov <bp@alien8.de> |
|---|---|
| Newsgroups | linux.kernel |
| Subject | [PATCH] x86/alternatives: Do not use sync_core() to serialize I$ |
| Date | 2016-12-03 16:10 +0100 |
| Message-ID | <sKk6R-2Xf-7@gated-at.bofh.it> (permalink) |
| References | (5 earlier) <sK1dU-7x5-5@gated-at.bofh.it> <sK1nz-7Px-3@gated-at.bofh.it> <sK1GV-7VR-27@gated-at.bofh.it> <sK1GV-7VR-25@gated-at.bofh.it> <sK1GW-7VR-29@gated-at.bofh.it> |
| Organization | linux.* mail to news gateway |
On Fri, Dec 02, 2016 at 08:28:44PM +0100, Borislav Petkov wrote:
> Ah, it is called only from apply_alternatives() but sure, it is safer
> this way. Lemme do that and run it through the boxes to see whether
> anything catches fire.
Looks good, ran it on a bunch of machines, two of them huge AMD and
Intel, 8-socket monsters.
Here is a version with a commit message:
---
From: Borislav Petkov <bp@suse.de>
Date: Fri, 2 Dec 2016 23:18:51 +0100
Subject: [PATCH] x86/alternatives: Do not use sync_core() to serialize I$
We use sync_core() in the alternatives code to stop speculative
execution of prefetched instructions because we are potentially changing
them and don't want to execute stale bytes.
What it does on most machines is call CPUID which is a serializing
instruction. And that's expensive.
However, the instruction cache is serialized when we're on the local CPU
and are changing the data through the same virtual address. So then, we
don't need the serializing CPUID but a simple control flow change. Last
being accomplished with a CALL/RET which the noinline causes.
So let's try it.
Signed-off-by: Borislav Petkov <bp@suse.de>
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andy Lutomirski <luto@amacapital.net>
---
arch/x86/kernel/alternative.c | 15 ++++++++++-----
1 file changed, 10 insertions(+), 5 deletions(-)
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 5cb272a7a5a3..c5b8f760473c 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -337,7 +337,11 @@ recompute_jump(struct alt_instr *a, u8 *orig_insn, u8 *repl_insn, u8 *insnbuf)
n_dspl, (unsigned long)orig_insn + n_dspl + repl_len);
}
-static void __init_or_module optimize_nops(struct alt_instr *a, u8 *instr)
+/*
+ * "noinline" to cause control flow change and thus invalidate I$ and
+ * cause refetch after modification.
+ */
+static void __init_or_module noinline optimize_nops(struct alt_instr *a, u8 *instr)
{
unsigned long flags;
@@ -346,7 +350,6 @@ static void __init_or_module optimize_nops(struct alt_instr *a, u8 *instr)
local_irq_save(flags);
add_nops(instr + (a->instrlen - a->padlen), a->padlen);
- sync_core();
local_irq_restore(flags);
DUMP_BYTES(instr, a->instrlen, "%p: [%d:%d) optimized NOPs: ",
@@ -359,9 +362,12 @@ static void __init_or_module optimize_nops(struct alt_instr *a, u8 *instr)
* This implies that asymmetric systems where APs have less capabilities than
* the boot processor are not handled. Tough. Make sure you disable such
* features by hand.
+ *
+ * Marked "noinline" to cause control flow change and thus insn cache
+ * to refetch changed I$ lines.
*/
-void __init_or_module apply_alternatives(struct alt_instr *start,
- struct alt_instr *end)
+void __init_or_module noinline apply_alternatives(struct alt_instr *start,
+ struct alt_instr *end)
{
struct alt_instr *a;
u8 *instr, *replacement;
@@ -667,7 +673,6 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode,
unsigned long flags;
local_irq_save(flags);
memcpy(addr, opcode, len);
- sync_core();
local_irq_restore(flags);
/* Could also do a CLFLUSH here to speed up CPU recovery; but
that causes hangs on some VIA CPUs. */
--
2.11.0
--
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.
Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread
[PATCH 0/4] CPUID-less CPU fixes and improvements Andy Lutomirski <luto@kernel.org> - 2016-11-30 21:40 +0100
[PATCH 4/4] x86/asm: Change sync_core() to use MOV to CR2 to serialize Andy Lutomirski <luto@kernel.org> - 2016-11-30 21:40 +0100
Re: [PATCH 4/4] x86/asm: Change sync_core() to use MOV to CR2 to serialize Borislav Petkov <bp@alien8.de> - 2016-12-01 10:30 +0100
Re: [PATCH 4/4] x86/asm: Change sync_core() to use MOV to CR2 to serialize Andy Lutomirski <luto@amacapital.net> - 2016-12-01 18:10 +0100
Re: [PATCH 4/4] x86/asm: Change sync_core() to use MOV to CR2 to serialize Andrew Cooper <andrew.cooper3@citrix.com> - 2016-12-01 18:50 +0100
Re: [PATCH 4/4] x86/asm: Change sync_core() to use MOV to CR2 to serialize Ingo Molnar <mingo@kernel.org> - 2016-12-02 08:40 +0100
[PATCH 3/4] x86/microcode/intel: Replace sync_core() with cpuid_eax(1) Andy Lutomirski <luto@kernel.org> - 2016-11-30 21:40 +0100
Re: [PATCH 3/4] x86/microcode/intel: Replace sync_core() with cpuid_eax(1) Borislav Petkov <bp@alien8.de> - 2016-12-01 10:20 +0100
[PATCH v2 6/6] x86/asm: Change sync_core() to use MOV to CR2 to serialize Andy Lutomirski <luto@kernel.org> - 2016-12-02 01:40 +0100
[PATCH v2 4/6] x86/paravirt: Make sync_core() be a paravirt op Andy Lutomirski <luto@kernel.org> - 2016-12-02 01:40 +0100
[PATCH v2 3/6] x86/microcode/intel: Replace sync_core() with cpuid_eax(1) Andy Lutomirski <luto@kernel.org> - 2016-12-02 01:40 +0100
[PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andy Lutomirski <luto@kernel.org> - 2016-12-02 01:40 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andrew Cooper <andrew.cooper3@citrix.com> - 2016-12-02 12:50 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andy Lutomirski <luto@amacapital.net> - 2016-12-02 18:10 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andrew Cooper <andrew.cooper3@citrix.com> - 2016-12-02 18:20 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andrew Cooper <andrew.cooper3@citrix.com> - 2016-12-02 18:30 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andy Lutomirski <luto@amacapital.net> - 2016-12-02 18:30 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Boris Ostrovsky <boris.ostrovsky@oracle.com> - 2016-12-02 19:50 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andy Lutomirski <luto@amacapital.net> - 2016-12-02 20:40 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Boris Ostrovsky <boris.ostrovsky@oracle.com> - 2016-12-02 21:10 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-02 18:40 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andy Lutomirski <luto@kernel.org> - 2016-12-02 18:40 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Borislav Petkov <bp@kernel.org> - 2016-12-02 19:20 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-02 19:30 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Borislav Petkov <bp@kernel.org> - 2016-12-02 20:00 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-02 20:10 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-02 20:30 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Borislav Petkov <bp@alien8.de> - 2016-12-02 20:30 +0100
[PATCH] x86/alternatives: Do not use sync_core() to serialize I$ Borislav Petkov <bp@alien8.de> - 2016-12-03 16:10 +0100
Re: [PATCH] x86/alternatives: Do not use sync_core() to serialize I$ Andy Lutomirski <luto@amacapital.net> - 2016-12-03 18:20 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-02 20:40 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andy Lutomirski <luto@amacapital.net> - 2016-12-02 21:50 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-02 22:20 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andy Lutomirski <luto@amacapital.net> - 2016-12-03 00:00 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-03 00:20 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andy Lutomirski <luto@amacapital.net> - 2016-12-02 20:40 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Andy Lutomirski <luto@amacapital.net> - 2016-12-02 20:30 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Borislav Petkov <bp@alien8.de> - 2016-12-02 20:40 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Borislav Petkov <bp@alien8.de> - 2016-12-02 20:30 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Linus Torvalds <torvalds@linux-foundation.org> - 2016-12-02 19:20 +0100
Re: [PATCH v2 5/6] x86/xen: Add a Xen-specific sync_core() implementation Borislav Petkov <bp@alien8.de> - 2016-12-03 13:50 +0100
[PATCH v2 2/6] Revert "x86/boot: Fail the boot if !M486 and CPUID is missing" Andy Lutomirski <luto@kernel.org> - 2016-12-02 01:40 +0100
Re: [PATCH 0/4] CPUID-less CPU fixes and improvements Andy Lutomirski <luto@amacapital.net> - 2016-12-02 01:40 +0100
Re: [PATCH 0/4] CPUID-less CPU fixes and improvements Ingo Molnar <mingo@kernel.org> - 2016-12-02 11:20 +0100
[PATCH v2 0/6] CPUID-less CPU/sync_core fixes and improvements Andy Lutomirski <luto@kernel.org> - 2016-12-02 01:40 +0100
[PATCH v2 1/6] x86/asm/32: Make sync_core() handle missing CPUID on all 32-bit kernels Andy Lutomirski <luto@kernel.org> - 2016-12-02 01:40 +0100
csiph-web