Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1706521 > unrolled thread
| Started by | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| First post | 2017-08-08 15:00 +0200 |
| Last post | 2017-08-08 15:00 +0200 |
| Articles | 3 — 1 participant |
Back to article view | Back to linux.kernel
[PATCHv4 00/14] Boot-time switching between 4- and 5-level paging "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2017-08-08 15:00 +0200
[PATCHv4 12/14] x86/mm: Allow to boot without la57 if CONFIG_X86_5LEVEL=y "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2017-08-08 15:00 +0200
[PATCHv4 03/14] x86/kasan: Use the same shadow offset for 4- and 5-level paging "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2017-08-08 15:00 +0200
| From | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| Date | 2017-08-08 15:00 +0200 |
| Subject | [PATCHv4 00/14] Boot-time switching between 4- and 5-level paging |
| Message-ID | <ucch3-8wk-3@gated-at.bofh.it> |
The basic idea is to implement the same logic as pgtable-nop4d.h provides, but at runtime. Runtime folding is only implemented for CONFIG_X86_5LEVEL=y case. With the option disabled, we do compile-time folding as before.. Initially, I tried to fold pgd instread. I've got to shell, but it required a lot of hacks as kernel threats pgd in a special way. Please review and consider applying. v4: - Use ALTERNATIVE to patch return_from_SYSCALL_64 (Andi); - Use __read_mostly where appropriate (Andi); - Make X86_5LEVEL dependant on SPARSEMEM_VMEMMAP; - Fix build errors and warnings; v3: - Make sparsemem data stuctures allocation dynamic to lower memory overhead on 4-level paging machines; - Allow XEN_PV and XEN_PVH to be enabled with X86_5LEVEL; - XEN cleanups; Kirill A. Shutemov (14): mm/sparsemem: Allocate mem_section at runtime for SPARSEMEM_EXTREME mm/zsmalloc: Prepare to variable MAX_PHYSMEM_BITS x86/kasan: Use the same shadow offset for 4- and 5-level paging x86/xen: Provide pre-built page tables only for XEN_PV and XEN_PVH x86/xen: Drop 5-level paging support code from XEN_PV code x86/boot/compressed/64: Detect and handle 5-level paging at boot-time x86/mm: Make virtual memory layout movable for CONFIG_X86_5LEVEL x86/mm: Make PGDIR_SHIFT and PTRS_PER_P4D variable x86/mm: Handle boot-time paging mode switching at early boot x86/mm: Fold p4d page table layer at runtime x86/mm: Replace compile-time checks for 5-level with runtime-time x86/mm: Allow to boot without la57 if CONFIG_X86_5LEVEL=y x86/xen: Allow XEN_PV and XEN_PVH to be enabled with X86_5LEVEL x86/mm: Offset boot-time paging mode switching cost Documentation/x86/x86_64/5level-paging.txt | 9 +- arch/x86/Kconfig | 6 +- arch/x86/boot/compressed/head_64.S | 24 ++++ arch/x86/boot/compressed/kaslr.c | 14 +++ arch/x86/boot/compressed/misc.h | 5 + arch/x86/entry/entry_64.S | 5 + arch/x86/include/asm/kaslr.h | 4 - arch/x86/include/asm/page_64.h | 4 + arch/x86/include/asm/page_64_types.h | 15 +-- arch/x86/include/asm/paravirt.h | 21 ++-- arch/x86/include/asm/pgalloc.h | 5 +- arch/x86/include/asm/pgtable.h | 10 +- arch/x86/include/asm/pgtable_32.h | 2 + arch/x86/include/asm/pgtable_32_types.h | 2 + arch/x86/include/asm/pgtable_64_types.h | 53 ++++++--- arch/x86/include/asm/pgtable_types.h | 67 +++-------- arch/x86/include/asm/processor.h | 2 +- arch/x86/include/asm/required-features.h | 8 +- arch/x86/include/asm/sparsemem.h | 9 +- arch/x86/kernel/Makefile | 3 +- arch/x86/kernel/head64.c | 73 ++++++++++-- arch/x86/kernel/head_64.S | 35 +++--- arch/x86/kernel/setup.c | 5 +- arch/x86/mm/dump_pagetables.c | 20 ++-- arch/x86/mm/fault.c | 2 +- arch/x86/mm/ident_map.c | 2 +- arch/x86/mm/init_64.c | 32 ++--- arch/x86/mm/kasan_init_64.c | 94 +++++++++++---- arch/x86/mm/kaslr.c | 27 ++--- arch/x86/platform/efi/efi_64.c | 6 +- arch/x86/power/hibernate_64.c | 6 +- arch/x86/xen/Kconfig | 5 - arch/x86/xen/mmu_pv.c | 180 +++++++++++++---------------- include/asm-generic/5level-fixup.h | 1 + include/asm-generic/pgtable-nop4d.h | 1 + include/linux/kasan.h | 2 +- include/linux/mmzone.h | 2 +- mm/kasan/kasan_init.c | 2 +- mm/page_alloc.c | 10 ++ mm/sparse.c | 17 ++- mm/zsmalloc.c | 6 + 41 files changed, 475 insertions(+), 321 deletions(-) -- 2.13.2
[toc] | [next] | [standalone]
| From | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| Date | 2017-08-08 15:00 +0200 |
| Subject | [PATCHv4 12/14] x86/mm: Allow to boot without la57 if CONFIG_X86_5LEVEL=y |
| Message-ID | <ucch6-8wk-47@gated-at.bofh.it> |
| In reply to | #1706521 |
All pieces of the puzzle are in place and we can now allow to boot with CONFIG_X86_5LEVEL=y on a machine without la57 support. Kernel will detect that la57 is missing and fold p4d at runtime. Update documentation and Kconfig option description to reflect the change. Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com> --- Documentation/x86/x86_64/5level-paging.txt | 9 +++------ arch/x86/Kconfig | 4 ++-- arch/x86/include/asm/required-features.h | 8 +------- 3 files changed, 6 insertions(+), 15 deletions(-) diff --git a/Documentation/x86/x86_64/5level-paging.txt b/Documentation/x86/x86_64/5level-paging.txt index 087251a0d99c..2432a5ef86d9 100644 --- a/Documentation/x86/x86_64/5level-paging.txt +++ b/Documentation/x86/x86_64/5level-paging.txt @@ -20,12 +20,9 @@ Documentation/x86/x86_64/mm.txt CONFIG_X86_5LEVEL=y enables the feature. -So far, a kernel compiled with the option enabled will be able to boot -only on machines that supports the feature -- see for 'la57' flag in -/proc/cpuinfo. - -The plan is to implement boot-time switching between 4- and 5-level paging -in the future. +Kernel with CONFIG_X86_5LEVEL=y still able to boot on 4-level hardware. +In this case additional page table level -- p4d -- will be folded at +runtime. == User-space and large virtual address space == diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index ac3358bb7bd2..939698570aa1 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -1409,8 +1409,8 @@ config X86_5LEVEL It will be supported by future Intel CPUs. - Note: a kernel with this option enabled can only be booted - on machines that support the feature. + A kernel with the option enabled can be booted on machines that + support 4- or 5-level paging. See Documentation/x86/x86_64/5level-paging.txt for more information. diff --git a/arch/x86/include/asm/required-features.h b/arch/x86/include/asm/required-features.h index d91ba04dd007..fac9a5c0abe9 100644 --- a/arch/x86/include/asm/required-features.h +++ b/arch/x86/include/asm/required-features.h @@ -53,12 +53,6 @@ # define NEED_MOVBE 0 #endif -#ifdef CONFIG_X86_5LEVEL -# define NEED_LA57 (1<<(X86_FEATURE_LA57 & 31)) -#else -# define NEED_LA57 0 -#endif - #ifdef CONFIG_X86_64 #ifdef CONFIG_PARAVIRT /* Paravirtualized systems may not have PSE or PGE available */ @@ -104,7 +98,7 @@ #define REQUIRED_MASK13 0 #define REQUIRED_MASK14 0 #define REQUIRED_MASK15 0 -#define REQUIRED_MASK16 (NEED_LA57) +#define REQUIRED_MASK16 0 #define REQUIRED_MASK17 0 #define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 18) -- 2.13.2
[toc] | [prev] | [next] | [standalone]
| From | "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> |
|---|---|
| Date | 2017-08-08 15:00 +0200 |
| Subject | [PATCHv4 03/14] x86/kasan: Use the same shadow offset for 4- and 5-level paging |
| Message-ID | <ucch5-8wk-43@gated-at.bofh.it> |
| In reply to | #1706521 |
We are going to support boot-time switching between 4- and 5-level
paging. For KASAN it means we cannot have different KASAN_SHADOW_OFFSET
for different paging modes: the constant is passed to gcc to generate
code and cannot be changed at runtime.
This patch changes KASAN code to use 0xdffffc0000000000 as shadow offset
for both 4- and 5-level paging.
For 5-level paging it means that shadow memory region is not aligned to
PGD boundary anymore and we have to handle unaligned parts of the region
properly.
In addition, we have to exclude paravirt code from KASAN instrumentation
as we now use set_pgd() before KASAN is fully ready.
Signed-off-by: Andrey Ryabinin <aryabinin@virtuozzo.com>
[kirill.shutemov@linux.intel.com: clenaup, changelog message]
Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
arch/x86/Kconfig | 1 -
arch/x86/kernel/Makefile | 3 +-
arch/x86/mm/kasan_init_64.c | 86 ++++++++++++++++++++++++++++++++++-----------
3 files changed, 67 insertions(+), 23 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index ce3ed304288d..2c9c4899d9ff 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -299,7 +299,6 @@ config ARCH_SUPPORTS_DEBUG_PAGEALLOC
config KASAN_SHADOW_OFFSET
hex
depends on KASAN
- default 0xdff8000000000000 if X86_5LEVEL
default 0xdffffc0000000000
config HAVE_INTEL_TXT
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 287eac7d207f..4509140232c3 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -24,7 +24,8 @@ endif
KASAN_SANITIZE_head$(BITS).o := n
KASAN_SANITIZE_dumpstack.o := n
KASAN_SANITIZE_dumpstack_$(BITS).o := n
-KASAN_SANITIZE_stacktrace.o := n
+KASAN_SANITIZE_stacktrace.o := n
+KASAN_SANITIZE_paravirt.o := n
OBJECT_FILES_NON_STANDARD_head_$(BITS).o := y
OBJECT_FILES_NON_STANDARD_relocate_kernel_$(BITS).o := y
diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c
index bc84b73684b7..f6b4db2647b5 100644
--- a/arch/x86/mm/kasan_init_64.c
+++ b/arch/x86/mm/kasan_init_64.c
@@ -15,6 +15,8 @@
extern struct range pfn_mapped[E820_MAX_ENTRIES];
+static p4d_t tmp_p4d_table[PTRS_PER_P4D] __initdata __aligned(PAGE_SIZE);
+
static int __init map_range(struct range *range)
{
unsigned long start;
@@ -30,8 +32,9 @@ static void __init clear_pgds(unsigned long start,
unsigned long end)
{
pgd_t *pgd;
+ unsigned long pgd_end = end & PGDIR_MASK;
- for (; start < end; start += PGDIR_SIZE) {
+ for (; start < pgd_end; start += PGDIR_SIZE) {
pgd = pgd_offset_k(start);
/*
* With folded p4d, pgd_clear() is nop, use p4d_clear()
@@ -42,29 +45,60 @@ static void __init clear_pgds(unsigned long start,
else
pgd_clear(pgd);
}
+
+ pgd = pgd_offset_k(start);
+ for (; start < end; start += P4D_SIZE)
+ p4d_clear(p4d_offset(pgd, start));
+}
+
+static inline p4d_t *early_p4d_offset(pgd_t *pgd, unsigned long addr)
+{
+ unsigned long p4d;
+
+ if (!IS_ENABLED(CONFIG_X86_5LEVEL))
+ return (p4d_t *)pgd;
+
+ p4d = __pa_nodebug(pgd_val(*pgd)) & PTE_PFN_MASK;
+ p4d += __START_KERNEL_map - phys_base;
+ return (p4d_t *)p4d + p4d_index(addr);
+}
+
+static void __init kasan_early_p4d_populate(pgd_t *pgd,
+ unsigned long addr,
+ unsigned long end)
+{
+ pgd_t pgd_entry;
+ p4d_t *p4d, p4d_entry;
+ unsigned long next;
+
+ if (pgd_none(*pgd)) {
+ pgd_entry = __pgd(_KERNPG_TABLE | __pa_nodebug(kasan_zero_p4d));
+ set_pgd(pgd, pgd_entry);
+ }
+
+ p4d = early_p4d_offset(pgd, addr);
+ do {
+ next = p4d_addr_end(addr, end);
+
+ if (!p4d_none(*p4d))
+ continue;
+
+ p4d_entry = __p4d(_KERNPG_TABLE | __pa_nodebug(kasan_zero_pud));
+ set_p4d(p4d, p4d_entry);
+ } while (p4d++, addr = next, addr != end && p4d_none(*p4d));
}
static void __init kasan_map_early_shadow(pgd_t *pgd)
{
- int i;
- unsigned long start = KASAN_SHADOW_START;
+ unsigned long addr = KASAN_SHADOW_START & PGDIR_MASK;
unsigned long end = KASAN_SHADOW_END;
+ unsigned long next;
- for (i = pgd_index(start); start < end; i++) {
- switch (CONFIG_PGTABLE_LEVELS) {
- case 4:
- pgd[i] = __pgd(__pa_nodebug(kasan_zero_pud) |
- _KERNPG_TABLE);
- break;
- case 5:
- pgd[i] = __pgd(__pa_nodebug(kasan_zero_p4d) |
- _KERNPG_TABLE);
- break;
- default:
- BUILD_BUG();
- }
- start += PGDIR_SIZE;
- }
+ pgd += pgd_index(addr);
+ do {
+ next = pgd_addr_end(addr, end);
+ kasan_early_p4d_populate(pgd, addr, next);
+ } while (pgd++, addr = next, addr != end);
}
#ifdef CONFIG_KASAN_INLINE
@@ -101,7 +135,7 @@ void __init kasan_early_init(void)
for (i = 0; i < PTRS_PER_PUD; i++)
kasan_zero_pud[i] = __pud(pud_val);
- for (i = 0; CONFIG_PGTABLE_LEVELS >= 5 && i < PTRS_PER_P4D; i++)
+ for (i = 0; IS_ENABLED(CONFIG_X86_5LEVEL) && i < PTRS_PER_P4D; i++)
kasan_zero_p4d[i] = __p4d(p4d_val);
kasan_map_early_shadow(early_top_pgt);
@@ -117,12 +151,22 @@ void __init kasan_init(void)
#endif
memcpy(early_top_pgt, init_top_pgt, sizeof(early_top_pgt));
+
+ if (IS_ENABLED(CONFIG_X86_5LEVEL)) {
+ void *ptr;
+
+ ptr = (void *)pgd_page_vaddr(*pgd_offset_k(KASAN_SHADOW_END));
+ memcpy(tmp_p4d_table, (void *)ptr, sizeof(tmp_p4d_table));
+ set_pgd(&early_top_pgt[pgd_index(KASAN_SHADOW_END)],
+ __pgd(__pa(tmp_p4d_table) | _KERNPG_TABLE));
+ }
+
load_cr3(early_top_pgt);
__flush_tlb_all();
- clear_pgds(KASAN_SHADOW_START, KASAN_SHADOW_END);
+ clear_pgds(KASAN_SHADOW_START & PGDIR_MASK, KASAN_SHADOW_END);
- kasan_populate_zero_shadow((void *)KASAN_SHADOW_START,
+ kasan_populate_zero_shadow((void *)(KASAN_SHADOW_START & PGDIR_MASK),
kasan_mem_to_shadow((void *)PAGE_OFFSET));
for (i = 0; i < E820_MAX_ENTRIES; i++) {
--
2.13.2
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web