Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1484745 > unrolled thread
| Started by | Christophe Leroy <christophe.leroy@c-s.fr> |
|---|---|
| First post | 2016-09-16 09:50 +0200 |
| Last post | 2016-09-20 07:30 +0200 |
| Articles | 13 — 3 participants |
Back to article view | Back to linux.kernel
[PATCH v2 0/3] powerpc: implementation of huge pages for 8xx Christophe Leroy <christophe.leroy@c-s.fr> - 2016-09-16 09:50 +0200
[PATCH v2 1/3] powerpc: port 64 bits pgtable_cache to 32 bits Christophe Leroy <christophe.leroy@c-s.fr> - 2016-09-16 09:50 +0200
Re: [PATCH v2 1/3] powerpc: port 64 bits pgtable_cache to 32 bits "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-09-19 07:30 +0200
Re: [PATCH v2 1/3] powerpc: port 64 bits pgtable_cache to 32 bits christophe leroy <christophe.leroy@c-s.fr> - 2016-09-19 20:50 +0200
[PATCH v2 2/3] powerpc: get hugetlbpage handling more generic Christophe Leroy <christophe.leroy@c-s.fr> - 2016-09-16 09:50 +0200
Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-09-19 07:50 +0200
Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic christophe leroy <christophe.leroy@c-s.fr> - 2016-09-19 20:40 +0200
Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-09-20 04:50 +0200
Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic Christophe Leroy <christophe.leroy@c-s.fr> - 2016-09-21 08:20 +0200
Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-09-19 08:00 +0200
Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic christophe leroy <christophe.leroy@c-s.fr> - 2016-09-19 20:40 +0200
Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> - 2016-09-20 04:30 +0200
Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic Christophe Leroy <christophe.leroy@c-s.fr> - 2016-09-20 07:30 +0200
| From | Christophe Leroy <christophe.leroy@c-s.fr> |
|---|---|
| Date | 2016-09-16 09:50 +0200 |
| Subject | [PATCH v2 0/3] powerpc: implementation of huge pages for 8xx |
| Message-ID | <shW4h-1Eu-7@gated-at.bofh.it> |
This is v2 of patch serie is the implementation of support of hugepages for the 8xx. v1 of the serie was including some other fixes and optimisations/reorganisations for the 8xx. Now the patch has been split and this part only focuses on the implementation of hugepages. Compared the v1, the last patch has been split in two parts. This patch serie applies on top of the patch serie named "Optimisation on 8xx prior to hugepage implementation" Christophe Leroy (3): powerpc: port 64 bits pgtable_cache to 32 bits powerpc: get hugetlbpage handling more generic powerpc/8xx: Implement support of hugepages arch/powerpc/include/asm/book3s/32/pgalloc.h | 44 +++++- arch/powerpc/include/asm/book3s/32/pgtable.h | 43 +++--- arch/powerpc/include/asm/book3s/64/pgtable.h | 3 - arch/powerpc/include/asm/hugetlb.h | 19 ++- arch/powerpc/include/asm/mmu-8xx.h | 35 +++++ arch/powerpc/include/asm/mmu.h | 23 +-- arch/powerpc/include/asm/nohash/32/pgalloc.h | 44 +++++- arch/powerpc/include/asm/nohash/32/pgtable.h | 45 +++--- arch/powerpc/include/asm/nohash/32/pte-8xx.h | 1 + arch/powerpc/include/asm/nohash/64/pgtable.h | 2 - arch/powerpc/include/asm/nohash/pgtable.h | 4 + arch/powerpc/include/asm/pgtable.h | 2 + arch/powerpc/include/asm/reg_8xx.h | 2 +- arch/powerpc/kernel/head_8xx.S | 119 ++++++++++++++- arch/powerpc/mm/Makefile | 3 +- arch/powerpc/mm/hugetlbpage.c | 212 ++++++++++++--------------- arch/powerpc/mm/init-common.c | 147 +++++++++++++++++++ arch/powerpc/mm/init_64.c | 77 ---------- arch/powerpc/mm/pgtable_32.c | 37 ----- arch/powerpc/mm/tlb_nohash.c | 21 ++- arch/powerpc/platforms/8xx/Kconfig | 1 + arch/powerpc/platforms/Kconfig.cputype | 1 + 22 files changed, 572 insertions(+), 313 deletions(-) create mode 100644 arch/powerpc/mm/init-common.c -- 2.1.0
[toc] | [next] | [standalone]
| From | Christophe Leroy <christophe.leroy@c-s.fr> |
|---|---|
| Date | 2016-09-16 09:50 +0200 |
| Subject | [PATCH v2 1/3] powerpc: port 64 bits pgtable_cache to 32 bits |
| Message-ID | <shW4h-1Eu-13@gated-at.bofh.it> |
| In reply to | #1484745 |
Today powerpc64 uses a set of pgtable_caches while powerpc32 uses
standard pages when using 4k pages and a single pgtable_cache
if using other size pages.
In preparation of implementing huge pages on the 8xx, this patch
replaces the specific powerpc32 handling by the 64 bits approach.
This is done by:
* moving 64 bits pgtable_cache_add() and pgtable_cache_init()
in a new file called init-common.c
* modifying pgtable_cache_init() to also handle the case
without PMD
* removing the 32 bits version of pgtable_cache_add() and
pgtable_cache_init()
* copying related header contents from 64 bits into both the
book3s/32 and nohash/32 header files
On the 8xx, the following cache sizes will be used:
* 4k pages mode:
- PGT_CACHE(10) for PGD
- PGT_CACHE(3) for 512k hugepage tables
* 16k pages mode:
- PGT_CACHE(6) for PGD
- PGT_CACHE(7) for 512k hugepage tables
- PGT_CACHE(3) for 8M hugepage tables
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
v2: in v1, hugepte_cache was wrongly replaced by PGT_CACHE(1).
This modification has been removed from v2.
arch/powerpc/include/asm/book3s/32/pgalloc.h | 44 ++++++--
arch/powerpc/include/asm/book3s/32/pgtable.h | 43 ++++----
arch/powerpc/include/asm/book3s/64/pgtable.h | 3 -
arch/powerpc/include/asm/nohash/32/pgalloc.h | 44 ++++++--
arch/powerpc/include/asm/nohash/32/pgtable.h | 45 ++++----
arch/powerpc/include/asm/nohash/64/pgtable.h | 2 -
arch/powerpc/include/asm/pgtable.h | 2 +
arch/powerpc/mm/Makefile | 3 +-
arch/powerpc/mm/init-common.c | 147 +++++++++++++++++++++++++++
arch/powerpc/mm/init_64.c | 77 --------------
arch/powerpc/mm/pgtable_32.c | 37 -------
11 files changed, 273 insertions(+), 174 deletions(-)
create mode 100644 arch/powerpc/mm/init-common.c
diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h
index 8e21bb4..d310546 100644
--- a/arch/powerpc/include/asm/book3s/32/pgalloc.h
+++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h
@@ -2,14 +2,42 @@
#define _ASM_POWERPC_BOOK3S_32_PGALLOC_H
#include <linux/threads.h>
+#include <linux/slab.h>
-/* For 32-bit, all levels of page tables are just drawn from get_free_page() */
-#define MAX_PGTABLE_INDEX_SIZE 0
+/*
+ * Functions that deal with pagetables that could be at any level of
+ * the table need to be passed an "index_size" so they know how to
+ * handle allocation. For PTE pages (which are linked to a struct
+ * page for now, and drawn from the main get_free_pages() pool), the
+ * allocation size will be (2^index_size * sizeof(pointer)) and
+ * allocations are drawn from the kmem_cache in PGT_CACHE(index_size).
+ *
+ * The maximum index size needs to be big enough to allow any
+ * pagetable sizes we need, but small enough to fit in the low bits of
+ * any page table pointer. In other words all pagetables, even tiny
+ * ones, must be aligned to allow at least enough low 0 bits to
+ * contain this value. This value is also used as a mask, so it must
+ * be one less than a power of two.
+ */
+#define MAX_PGTABLE_INDEX_SIZE 0xf
extern void __bad_pte(pmd_t *pmd);
-extern pgd_t *pgd_alloc(struct mm_struct *mm);
-extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
+extern struct kmem_cache *pgtable_cache[];
+#define PGT_CACHE(shift) ({ \
+ BUG_ON(!(shift)); \
+ pgtable_cache[(shift) - 1]; \
+ })
+
+static inline pgd_t *pgd_alloc(struct mm_struct *mm)
+{
+ return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL);
+}
+
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
+{
+ kmem_cache_free(PGT_CACHE(PGD_INDEX_SIZE), pgd);
+}
/*
* We don't have any real pmd's, and this code never triggers because
@@ -68,8 +96,12 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage)
static inline void pgtable_free(void *table, unsigned index_size)
{
- BUG_ON(index_size); /* 32-bit doesn't use this */
- free_page((unsigned long)table);
+ if (!index_size) {
+ free_page((unsigned long)table);
+ } else {
+ BUG_ON(index_size > MAX_PGTABLE_INDEX_SIZE);
+ kmem_cache_free(PGT_CACHE(index_size), table);
+ }
}
#define check_pgt_cache() do { } while (0)
diff --git a/arch/powerpc/include/asm/book3s/32/pgtable.h b/arch/powerpc/include/asm/book3s/32/pgtable.h
index 6b8b2d5..f887499 100644
--- a/arch/powerpc/include/asm/book3s/32/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/32/pgtable.h
@@ -8,6 +8,26 @@
/* And here we include common definitions */
#include <asm/pte-common.h>
+#define PTE_INDEX_SIZE PTE_SHIFT
+#define PMD_INDEX_SIZE 0
+#define PUD_INDEX_SIZE 0
+#define PGD_INDEX_SIZE (32 - PGDIR_SHIFT)
+
+#define PMD_CACHE_INDEX PMD_INDEX_SIZE
+
+#ifndef __ASSEMBLY__
+#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE)
+#define PMD_TABLE_SIZE (sizeof(pmd_t) << PTE_INDEX_SIZE)
+#define PUD_TABLE_SIZE (sizeof(pud_t) << PTE_INDEX_SIZE)
+#define PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE)
+#endif /* __ASSEMBLY__ */
+
+#define PTRS_PER_PTE (1 << PTE_INDEX_SIZE)
+#define PTRS_PER_PGD (1 << PGD_INDEX_SIZE)
+
+/* With 4k base page size, hugepage PTEs go at the PMD level */
+#define MIN_HUGEPTE_SHIFT PMD_SHIFT
+
/*
* The normal case is that PTEs are 32-bits and we have a 1-page
* 1024-entry pgdir pointing to 1-page 1024-entry PTE pages. -- paulus
@@ -19,14 +39,10 @@
* -Matt
*/
/* PGDIR_SHIFT determines what a top-level page table entry can map */
-#define PGDIR_SHIFT (PAGE_SHIFT + PTE_SHIFT)
+#define PGDIR_SHIFT (PAGE_SHIFT + PTE_INDEX_SIZE)
#define PGDIR_SIZE (1UL << PGDIR_SHIFT)
#define PGDIR_MASK (~(PGDIR_SIZE-1))
-#define PTRS_PER_PTE (1 << PTE_SHIFT)
-#define PTRS_PER_PMD 1
-#define PTRS_PER_PGD (1 << (32 - PGDIR_SHIFT))
-
#define USER_PTRS_PER_PGD (TASK_SIZE / PGDIR_SIZE)
/*
* This is the bottom of the PKMAP area with HIGHMEM or an arbitrary
@@ -82,12 +98,8 @@
extern unsigned long ioremap_bot;
-/*
- * entries per page directory level: our page-table tree is two-level, so
- * we don't really have any PMD directory.
- */
-#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_SHIFT)
-#define PGD_TABLE_SIZE (sizeof(pgd_t) << (32 - PGDIR_SHIFT))
+/* Bits to mask out from a PGD to get to the PUD page */
+#define PGD_MASKED_BITS 0
#define pte_ERROR(e) \
pr_err("%s:%d: bad pte %llx.\n", __FILE__, __LINE__, \
@@ -283,15 +295,6 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 3 })
#define __swp_entry_to_pte(x) ((pte_t) { (x).val << 3 })
-#ifndef CONFIG_PPC_4K_PAGES
-void pgtable_cache_init(void);
-#else
-/*
- * No page table caches to initialise
- */
-#define pgtable_cache_init() do { } while (0)
-#endif
-
extern int get_pteptr(struct mm_struct *mm, unsigned long addr, pte_t **ptep,
pmd_t **pmdp);
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index 9fd77f8..0a46a5f 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -789,9 +789,6 @@ extern struct page *pgd_page(pgd_t pgd);
#define pgd_ERROR(e) \
pr_err("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
-void pgtable_cache_add(unsigned shift, void (*ctor)(void *));
-void pgtable_cache_init(void);
-
static inline int map_kernel_page(unsigned long ea, unsigned long pa,
unsigned long flags)
{
diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/include/asm/nohash/32/pgalloc.h
index 76d6b9e..6331392 100644
--- a/arch/powerpc/include/asm/nohash/32/pgalloc.h
+++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h
@@ -2,14 +2,42 @@
#define _ASM_POWERPC_PGALLOC_32_H
#include <linux/threads.h>
+#include <linux/slab.h>
-/* For 32-bit, all levels of page tables are just drawn from get_free_page() */
-#define MAX_PGTABLE_INDEX_SIZE 0
+/*
+ * Functions that deal with pagetables that could be at any level of
+ * the table need to be passed an "index_size" so they know how to
+ * handle allocation. For PTE pages (which are linked to a struct
+ * page for now, and drawn from the main get_free_pages() pool), the
+ * allocation size will be (2^index_size * sizeof(pointer)) and
+ * allocations are drawn from the kmem_cache in PGT_CACHE(index_size).
+ *
+ * The maximum index size needs to be big enough to allow any
+ * pagetable sizes we need, but small enough to fit in the low bits of
+ * any page table pointer. In other words all pagetables, even tiny
+ * ones, must be aligned to allow at least enough low 0 bits to
+ * contain this value. This value is also used as a mask, so it must
+ * be one less than a power of two.
+ */
+#define MAX_PGTABLE_INDEX_SIZE 0xf
extern void __bad_pte(pmd_t *pmd);
-extern pgd_t *pgd_alloc(struct mm_struct *mm);
-extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
+extern struct kmem_cache *pgtable_cache[];
+#define PGT_CACHE(shift) ({ \
+ BUG_ON(!(shift)); \
+ pgtable_cache[(shift) - 1]; \
+ })
+
+static inline pgd_t *pgd_alloc(struct mm_struct *mm)
+{
+ return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL);
+}
+
+static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
+{
+ kmem_cache_free(PGT_CACHE(PGD_INDEX_SIZE), pgd);
+}
/*
* We don't have any real pmd's, and this code never triggers because
@@ -68,8 +96,12 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage)
static inline void pgtable_free(void *table, unsigned index_size)
{
- BUG_ON(index_size); /* 32-bit doesn't use this */
- free_page((unsigned long)table);
+ if (!index_size) {
+ free_page((unsigned long)table);
+ } else {
+ BUG_ON(index_size > MAX_PGTABLE_INDEX_SIZE);
+ kmem_cache_free(PGT_CACHE(index_size), table);
+ }
}
#define check_pgt_cache() do { } while (0)
diff --git a/arch/powerpc/include/asm/nohash/32/pgtable.h b/arch/powerpc/include/asm/nohash/32/pgtable.h
index c219ef7..8cbe222 100644
--- a/arch/powerpc/include/asm/nohash/32/pgtable.h
+++ b/arch/powerpc/include/asm/nohash/32/pgtable.h
@@ -16,6 +16,26 @@ extern int icache_44x_need_flush;
#endif /* __ASSEMBLY__ */
+#define PTE_INDEX_SIZE PTE_SHIFT
+#define PMD_INDEX_SIZE 0
+#define PUD_INDEX_SIZE 0
+#define PGD_INDEX_SIZE (32 - PGDIR_SHIFT)
+
+#define PMD_CACHE_INDEX PMD_INDEX_SIZE
+
+#ifndef __ASSEMBLY__
+#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE)
+#define PMD_TABLE_SIZE (sizeof(pmd_t) << PTE_INDEX_SIZE)
+#define PUD_TABLE_SIZE (sizeof(pud_t) << PTE_INDEX_SIZE)
+#define PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE)
+#endif /* __ASSEMBLY__ */
+
+#define PTRS_PER_PTE (1 << PTE_INDEX_SIZE)
+#define PTRS_PER_PGD (1 << PGD_INDEX_SIZE)
+
+/* With 4k base page size, hugepage PTEs go at the PMD level */
+#define MIN_HUGEPTE_SHIFT PMD_SHIFT
+
/*
* The normal case is that PTEs are 32-bits and we have a 1-page
* 1024-entry pgdir pointing to 1-page 1024-entry PTE pages. -- paulus
@@ -27,22 +47,12 @@ extern int icache_44x_need_flush;
* -Matt
*/
/* PGDIR_SHIFT determines what a top-level page table entry can map */
-#define PGDIR_SHIFT (PAGE_SHIFT + PTE_SHIFT)
+#define PGDIR_SHIFT (PAGE_SHIFT + PTE_INDEX_SIZE)
#define PGDIR_SIZE (1UL << PGDIR_SHIFT)
#define PGDIR_MASK (~(PGDIR_SIZE-1))
-/*
- * entries per page directory level: our page-table tree is two-level, so
- * we don't really have any PMD directory.
- */
-#ifndef __ASSEMBLY__
-#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_SHIFT)
-#define PGD_TABLE_SIZE (sizeof(pgd_t) << (32 - PGDIR_SHIFT))
-#endif /* __ASSEMBLY__ */
-
-#define PTRS_PER_PTE (1 << PTE_SHIFT)
-#define PTRS_PER_PMD 1
-#define PTRS_PER_PGD (1 << (32 - PGDIR_SHIFT))
+/* Bits to mask out from a PGD to get to the PUD page */
+#define PGD_MASKED_BITS 0
#define USER_PTRS_PER_PGD (TASK_SIZE / PGDIR_SIZE)
#define FIRST_USER_ADDRESS 0UL
@@ -328,15 +338,6 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 3 })
#define __swp_entry_to_pte(x) ((pte_t) { (x).val << 3 })
-#ifndef CONFIG_PPC_4K_PAGES
-void pgtable_cache_init(void);
-#else
-/*
- * No page table caches to initialise
- */
-#define pgtable_cache_init() do { } while (0)
-#endif
-
extern int get_pteptr(struct mm_struct *mm, unsigned long addr, pte_t **ptep,
pmd_t **pmdp);
diff --git a/arch/powerpc/include/asm/nohash/64/pgtable.h b/arch/powerpc/include/asm/nohash/64/pgtable.h
index 653a183..619018a 100644
--- a/arch/powerpc/include/asm/nohash/64/pgtable.h
+++ b/arch/powerpc/include/asm/nohash/64/pgtable.h
@@ -358,8 +358,6 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val((pte)) })
#define __swp_entry_to_pte(x) __pte((x).val)
-void pgtable_cache_add(unsigned shift, void (*ctor)(void *));
-void pgtable_cache_init(void);
extern int map_kernel_page(unsigned long ea, unsigned long pa,
unsigned long flags);
extern int __meminit vmemmap_create_mapping(unsigned long start,
diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h
index 9bd87f2..dd01212 100644
--- a/arch/powerpc/include/asm/pgtable.h
+++ b/arch/powerpc/include/asm/pgtable.h
@@ -78,6 +78,8 @@ static inline pte_t *find_linux_pte_or_hugepte(pgd_t *pgdir, unsigned long ea,
unsigned long vmalloc_to_phys(void *vmalloc_addr);
+void pgtable_cache_add(unsigned shift, void (*ctor)(void *));
+void pgtable_cache_init(void);
#endif /* __ASSEMBLY__ */
#endif /* _ASM_POWERPC_PGTABLE_H */
diff --git a/arch/powerpc/mm/Makefile b/arch/powerpc/mm/Makefile
index 1a4e570..e8a86d2 100644
--- a/arch/powerpc/mm/Makefile
+++ b/arch/powerpc/mm/Makefile
@@ -7,7 +7,8 @@ subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
ccflags-$(CONFIG_PPC64) := $(NO_MINIMAL_TOC)
obj-y := fault.o mem.o pgtable.o mmap.o \
- init_$(BITS).o pgtable_$(BITS).o
+ init_$(BITS).o pgtable_$(BITS).o \
+ init-common.o
obj-$(CONFIG_PPC_MMU_NOHASH) += mmu_context_nohash.o tlb_nohash.o \
tlb_nohash_low.o
obj-$(CONFIG_PPC_BOOK3E) += tlb_low_$(BITS)e.o
diff --git a/arch/powerpc/mm/init-common.c b/arch/powerpc/mm/init-common.c
new file mode 100644
index 0000000..ab2b947
--- /dev/null
+++ b/arch/powerpc/mm/init-common.c
@@ -0,0 +1,147 @@
+/*
+ * PowerPC version
+ * Copyright (C) 1995-1996 Gary Thomas (gdt@linuxppc.org)
+ *
+ * Modifications by Paul Mackerras (PowerMac) (paulus@cs.anu.edu.au)
+ * and Cort Dougan (PReP) (cort@cs.nmt.edu)
+ * Copyright (C) 1996 Paul Mackerras
+ *
+ * Derived from "arch/i386/mm/init.c"
+ * Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
+ *
+ * Dave Engebretsen <engebret@us.ibm.com>
+ * Rework for PPC64 port.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ *
+ */
+
+#undef DEBUG
+
+#include <linux/signal.h>
+#include <linux/sched.h>
+#include <linux/kernel.h>
+#include <linux/errno.h>
+#include <linux/string.h>
+#include <linux/types.h>
+#include <linux/mman.h>
+#include <linux/mm.h>
+#include <linux/swap.h>
+#include <linux/stddef.h>
+#include <linux/vmalloc.h>
+#include <linux/init.h>
+#include <linux/delay.h>
+#include <linux/highmem.h>
+#include <linux/idr.h>
+#include <linux/nodemask.h>
+#include <linux/module.h>
+#include <linux/poison.h>
+#include <linux/memblock.h>
+#include <linux/hugetlb.h>
+#include <linux/slab.h>
+
+#include <asm/pgalloc.h>
+#include <asm/page.h>
+#include <asm/prom.h>
+#include <asm/rtas.h>
+#include <asm/io.h>
+#include <asm/mmu_context.h>
+#include <asm/pgtable.h>
+#include <asm/mmu.h>
+#include <asm/uaccess.h>
+#include <asm/smp.h>
+#include <asm/machdep.h>
+#include <asm/tlb.h>
+#include <asm/eeh.h>
+#include <asm/processor.h>
+#include <asm/mmzone.h>
+#include <asm/cputable.h>
+#include <asm/sections.h>
+#include <asm/iommu.h>
+#include <asm/vdso.h>
+
+#include "mmu_decl.h"
+
+static void pgd_ctor(void *addr)
+{
+ memset(addr, 0, PGD_TABLE_SIZE);
+}
+
+static void pud_ctor(void *addr)
+{
+ memset(addr, 0, PUD_TABLE_SIZE);
+}
+
+static void pmd_ctor(void *addr)
+{
+ memset(addr, 0, PMD_TABLE_SIZE);
+}
+
+struct kmem_cache *pgtable_cache[MAX_PGTABLE_INDEX_SIZE];
+
+/*
+ * Create a kmem_cache() for pagetables. This is not used for PTE
+ * pages - they're linked to struct page, come from the normal free
+ * pages pool and have a different entry size (see real_pte_t) to
+ * everything else. Caches created by this function are used for all
+ * the higher level pagetables, and for hugepage pagetables.
+ */
+void pgtable_cache_add(unsigned shift, void (*ctor)(void *))
+{
+ char *name;
+ unsigned long table_size = sizeof(void *) << shift;
+ unsigned long align = table_size;
+
+ /* When batching pgtable pointers for RCU freeing, we store
+ * the index size in the low bits. Table alignment must be
+ * big enough to fit it.
+ *
+ * Likewise, hugeapge pagetable pointers contain a (different)
+ * shift value in the low bits. All tables must be aligned so
+ * as to leave enough 0 bits in the address to contain it. */
+ unsigned long minalign = max(MAX_PGTABLE_INDEX_SIZE + 1,
+ HUGEPD_SHIFT_MASK + 1);
+ struct kmem_cache *new;
+
+ /* It would be nice if this was a BUILD_BUG_ON(), but at the
+ * moment, gcc doesn't seem to recognize is_power_of_2 as a
+ * constant expression, so so much for that. */
+ BUG_ON(!is_power_of_2(minalign));
+ BUG_ON((shift < 1) || (shift > MAX_PGTABLE_INDEX_SIZE));
+
+ if (PGT_CACHE(shift))
+ return; /* Already have a cache of this size */
+
+ align = max_t(unsigned long, align, minalign);
+ name = kasprintf(GFP_KERNEL, "pgtable-2^%d", shift);
+ new = kmem_cache_create(name, table_size, align, 0, ctor);
+ kfree(name);
+ pgtable_cache[shift - 1] = new;
+ pr_debug("Allocated pgtable cache for order %d\n", shift);
+}
+
+
+void pgtable_cache_init(void)
+{
+ pgtable_cache_add(PGD_INDEX_SIZE, pgd_ctor);
+
+ if (PMD_INDEX_SIZE && !PGT_CACHE(PMD_INDEX_SIZE))
+ pgtable_cache_add(PMD_CACHE_INDEX, pmd_ctor);
+ /*
+ * In all current configs, when the PUD index exists it's the
+ * same size as either the pgd or pmd index except with THP enabled
+ * on book3s 64
+ */
+ if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
+ pgtable_cache_add(PUD_INDEX_SIZE, pud_ctor);
+
+ if (!PGT_CACHE(PGD_INDEX_SIZE))
+ panic("Couldn't allocate pgd cache");
+ if (PMD_INDEX_SIZE && !PGT_CACHE(PMD_INDEX_SIZE))
+ panic("Couldn't allocate pmd pgtable caches");
+ if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
+ panic("Couldn't allocate pud pgtable caches");
+}
diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
index 16ada1e..a000c35 100644
--- a/arch/powerpc/mm/init_64.c
+++ b/arch/powerpc/mm/init_64.c
@@ -80,83 +80,6 @@ EXPORT_SYMBOL_GPL(memstart_addr);
phys_addr_t kernstart_addr;
EXPORT_SYMBOL_GPL(kernstart_addr);
-static void pgd_ctor(void *addr)
-{
- memset(addr, 0, PGD_TABLE_SIZE);
-}
-
-static void pud_ctor(void *addr)
-{
- memset(addr, 0, PUD_TABLE_SIZE);
-}
-
-static void pmd_ctor(void *addr)
-{
- memset(addr, 0, PMD_TABLE_SIZE);
-}
-
-struct kmem_cache *pgtable_cache[MAX_PGTABLE_INDEX_SIZE];
-
-/*
- * Create a kmem_cache() for pagetables. This is not used for PTE
- * pages - they're linked to struct page, come from the normal free
- * pages pool and have a different entry size (see real_pte_t) to
- * everything else. Caches created by this function are used for all
- * the higher level pagetables, and for hugepage pagetables.
- */
-void pgtable_cache_add(unsigned shift, void (*ctor)(void *))
-{
- char *name;
- unsigned long table_size = sizeof(void *) << shift;
- unsigned long align = table_size;
-
- /* When batching pgtable pointers for RCU freeing, we store
- * the index size in the low bits. Table alignment must be
- * big enough to fit it.
- *
- * Likewise, hugeapge pagetable pointers contain a (different)
- * shift value in the low bits. All tables must be aligned so
- * as to leave enough 0 bits in the address to contain it. */
- unsigned long minalign = max(MAX_PGTABLE_INDEX_SIZE + 1,
- HUGEPD_SHIFT_MASK + 1);
- struct kmem_cache *new;
-
- /* It would be nice if this was a BUILD_BUG_ON(), but at the
- * moment, gcc doesn't seem to recognize is_power_of_2 as a
- * constant expression, so so much for that. */
- BUG_ON(!is_power_of_2(minalign));
- BUG_ON((shift < 1) || (shift > MAX_PGTABLE_INDEX_SIZE));
-
- if (PGT_CACHE(shift))
- return; /* Already have a cache of this size */
-
- align = max_t(unsigned long, align, minalign);
- name = kasprintf(GFP_KERNEL, "pgtable-2^%d", shift);
- new = kmem_cache_create(name, table_size, align, 0, ctor);
- kfree(name);
- pgtable_cache[shift - 1] = new;
- pr_debug("Allocated pgtable cache for order %d\n", shift);
-}
-
-
-void pgtable_cache_init(void)
-{
- pgtable_cache_add(PGD_INDEX_SIZE, pgd_ctor);
- pgtable_cache_add(PMD_CACHE_INDEX, pmd_ctor);
- /*
- * In all current configs, when the PUD index exists it's the
- * same size as either the pgd or pmd index except with THP enabled
- * on book3s 64
- */
- if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
- pgtable_cache_add(PUD_INDEX_SIZE, pud_ctor);
-
- if (!PGT_CACHE(PGD_INDEX_SIZE) || !PGT_CACHE(PMD_CACHE_INDEX))
- panic("Couldn't allocate pgtable caches");
- if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
- panic("Couldn't allocate pud pgtable caches");
-}
-
#ifdef CONFIG_SPARSEMEM_VMEMMAP
/*
* Given an address within the vmemmap, determine the pfn of the page that
diff --git a/arch/powerpc/mm/pgtable_32.c b/arch/powerpc/mm/pgtable_32.c
index 0ae0572..a65c0b4 100644
--- a/arch/powerpc/mm/pgtable_32.c
+++ b/arch/powerpc/mm/pgtable_32.c
@@ -42,43 +42,6 @@ EXPORT_SYMBOL(ioremap_bot); /* aka VMALLOC_END */
extern char etext[], _stext[], _sinittext[], _einittext[];
-#define PGDIR_ORDER (32 + PGD_T_LOG2 - PGDIR_SHIFT)
-
-#ifndef CONFIG_PPC_4K_PAGES
-static struct kmem_cache *pgtable_cache;
-
-void pgtable_cache_init(void)
-{
- pgtable_cache = kmem_cache_create("PGDIR cache", 1 << PGDIR_ORDER,
- 1 << PGDIR_ORDER, 0, NULL);
- if (pgtable_cache == NULL)
- panic("Couldn't allocate pgtable caches");
-}
-#endif
-
-pgd_t *pgd_alloc(struct mm_struct *mm)
-{
- pgd_t *ret;
-
- /* pgdir take page or two with 4K pages and a page fraction otherwise */
-#ifndef CONFIG_PPC_4K_PAGES
- ret = kmem_cache_alloc(pgtable_cache, GFP_KERNEL | __GFP_ZERO);
-#else
- ret = (pgd_t *)__get_free_pages(GFP_KERNEL|__GFP_ZERO,
- PGDIR_ORDER - PAGE_SHIFT);
-#endif
- return ret;
-}
-
-void pgd_free(struct mm_struct *mm, pgd_t *pgd)
-{
-#ifndef CONFIG_PPC_4K_PAGES
- kmem_cache_free(pgtable_cache, (void *)pgd);
-#else
- free_pages((unsigned long)pgd, PGDIR_ORDER - PAGE_SHIFT);
-#endif
-}
-
__ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
{
pte_t *pte;
--
2.1.0
[toc] | [prev] | [next] | [standalone]
| From | "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> |
|---|---|
| Date | 2016-09-19 07:30 +0200 |
| Subject | Re: [PATCH v2 1/3] powerpc: port 64 bits pgtable_cache to 32 bits |
| Message-ID | <siZjs-1jp-7@gated-at.bofh.it> |
| In reply to | #1484746 |
Christophe Leroy <christophe.leroy@c-s.fr> writes:
> Today powerpc64 uses a set of pgtable_caches while powerpc32 uses
> standard pages when using 4k pages and a single pgtable_cache
> if using other size pages.
>
> In preparation of implementing huge pages on the 8xx, this patch
> replaces the specific powerpc32 handling by the 64 bits approach.
>
> This is done by:
> * moving 64 bits pgtable_cache_add() and pgtable_cache_init()
> in a new file called init-common.c
> * modifying pgtable_cache_init() to also handle the case
> without PMD
> * removing the 32 bits version of pgtable_cache_add() and
> pgtable_cache_init()
> * copying related header contents from 64 bits into both the
> book3s/32 and nohash/32 header files
>
> On the 8xx, the following cache sizes will be used:
> * 4k pages mode:
> - PGT_CACHE(10) for PGD
> - PGT_CACHE(3) for 512k hugepage tables
> * 16k pages mode:
> - PGT_CACHE(6) for PGD
> - PGT_CACHE(7) for 512k hugepage tables
> - PGT_CACHE(3) for 8M hugepage tables
>
> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
> ---
> v2: in v1, hugepte_cache was wrongly replaced by PGT_CACHE(1).
> This modification has been removed from v2.
>
> arch/powerpc/include/asm/book3s/32/pgalloc.h | 44 ++++++--
> arch/powerpc/include/asm/book3s/32/pgtable.h | 43 ++++----
> arch/powerpc/include/asm/book3s/64/pgtable.h | 3 -
> arch/powerpc/include/asm/nohash/32/pgalloc.h | 44 ++++++--
> arch/powerpc/include/asm/nohash/32/pgtable.h | 45 ++++----
> arch/powerpc/include/asm/nohash/64/pgtable.h | 2 -
> arch/powerpc/include/asm/pgtable.h | 2 +
> arch/powerpc/mm/Makefile | 3 +-
> arch/powerpc/mm/init-common.c | 147 +++++++++++++++++++++++++++
> arch/powerpc/mm/init_64.c | 77 --------------
> arch/powerpc/mm/pgtable_32.c | 37 -------
> 11 files changed, 273 insertions(+), 174 deletions(-)
> create mode 100644 arch/powerpc/mm/init-common.c
>
> diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h
> index 8e21bb4..d310546 100644
> --- a/arch/powerpc/include/asm/book3s/32/pgalloc.h
> +++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h
> @@ -2,14 +2,42 @@
> #define _ASM_POWERPC_BOOK3S_32_PGALLOC_H
>
> #include <linux/threads.h>
> +#include <linux/slab.h>
>
> -/* For 32-bit, all levels of page tables are just drawn from get_free_page() */
> -#define MAX_PGTABLE_INDEX_SIZE 0
> +/*
> + * Functions that deal with pagetables that could be at any level of
> + * the table need to be passed an "index_size" so they know how to
> + * handle allocation. For PTE pages (which are linked to a struct
> + * page for now, and drawn from the main get_free_pages() pool), the
> + * allocation size will be (2^index_size * sizeof(pointer)) and
> + * allocations are drawn from the kmem_cache in PGT_CACHE(index_size).
> + *
> + * The maximum index size needs to be big enough to allow any
> + * pagetable sizes we need, but small enough to fit in the low bits of
> + * any page table pointer. In other words all pagetables, even tiny
> + * ones, must be aligned to allow at least enough low 0 bits to
> + * contain this value. This value is also used as a mask, so it must
> + * be one less than a power of two.
> + */
> +#define MAX_PGTABLE_INDEX_SIZE 0xf
>
> extern void __bad_pte(pmd_t *pmd);
>
> -extern pgd_t *pgd_alloc(struct mm_struct *mm);
> -extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
> +extern struct kmem_cache *pgtable_cache[];
> +#define PGT_CACHE(shift) ({ \
> + BUG_ON(!(shift)); \
> + pgtable_cache[(shift) - 1]; \
> + })
> +
> +static inline pgd_t *pgd_alloc(struct mm_struct *mm)
> +{
> + return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL);
> +}
> +
> +static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
> +{
> + kmem_cache_free(PGT_CACHE(PGD_INDEX_SIZE), pgd);
> +}
>
> /*
> * We don't have any real pmd's, and this code never triggers because
> @@ -68,8 +96,12 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage)
>
> static inline void pgtable_free(void *table, unsigned index_size)
> {
> - BUG_ON(index_size); /* 32-bit doesn't use this */
> - free_page((unsigned long)table);
> + if (!index_size) {
> + free_page((unsigned long)table);
> + } else {
> + BUG_ON(index_size > MAX_PGTABLE_INDEX_SIZE);
> + kmem_cache_free(PGT_CACHE(index_size), table);
> + }
> }
>
> #define check_pgt_cache() do { } while (0)
> diff --git a/arch/powerpc/include/asm/book3s/32/pgtable.h b/arch/powerpc/include/asm/book3s/32/pgtable.h
> index 6b8b2d5..f887499 100644
> --- a/arch/powerpc/include/asm/book3s/32/pgtable.h
> +++ b/arch/powerpc/include/asm/book3s/32/pgtable.h
> @@ -8,6 +8,26 @@
> /* And here we include common definitions */
> #include <asm/pte-common.h>
>
> +#define PTE_INDEX_SIZE PTE_SHIFT
> +#define PMD_INDEX_SIZE 0
> +#define PUD_INDEX_SIZE 0
> +#define PGD_INDEX_SIZE (32 - PGDIR_SHIFT)
> +
> +#define PMD_CACHE_INDEX PMD_INDEX_SIZE
> +
> +#ifndef __ASSEMBLY__
> +#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE)
> +#define PMD_TABLE_SIZE (sizeof(pmd_t) << PTE_INDEX_SIZE)
> +#define PUD_TABLE_SIZE (sizeof(pud_t) << PTE_INDEX_SIZE)
> +#define PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE)
> +#endif /* __ASSEMBLY__ */
Are these table size correct ? IIUC, We will have only PGD and PTE
tables right ?
> +
> +#define PTRS_PER_PTE (1 << PTE_INDEX_SIZE)
> +#define PTRS_PER_PGD (1 << PGD_INDEX_SIZE)
> +
> +/* With 4k base page size, hugepage PTEs go at the PMD level */
> +#define MIN_HUGEPTE_SHIFT PMD_SHIFT
> +
What does that comment mean ? I guess that came from copy-paste from
other headers. I am not sure what it means there either other than the
64k hash config, where we place hugepage PTE at the PMD level. (ie, no hugepd).
> /*
> * The normal case is that PTEs are 32-bits and we have a 1-page
> * 1024-entry pgdir pointing to 1-page 1024-entry PTE pages. -- paulus
> @@ -19,14 +39,10 @@
> * -Matt
> */
> /* PGDIR_SHIFT determines what a top-level page table entry can map */
> -#define PGDIR_SHIFT (PAGE_SHIFT + PTE_SHIFT)
> +#define PGDIR_SHIFT (PAGE_SHIFT + PTE_INDEX_SIZE)
> #define PGDIR_SIZE (1UL << PGDIR_SHIFT)
> #define PGDIR_MASK (~(PGDIR_SIZE-1))
>
> -#define PTRS_PER_PTE (1 << PTE_SHIFT)
> -#define PTRS_PER_PMD 1
> -#define PTRS_PER_PGD (1 << (32 - PGDIR_SHIFT))
> -
> #define USER_PTRS_PER_PGD (TASK_SIZE / PGDIR_SIZE)
> /*
> * This is the bottom of the PKMAP area with HIGHMEM or an arbitrary
> @@ -82,12 +98,8 @@
>
> extern unsigned long ioremap_bot;
>
> -/*
> - * entries per page directory level: our page-table tree is two-level, so
> - * we don't really have any PMD directory.
> - */
> -#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_SHIFT)
> -#define PGD_TABLE_SIZE (sizeof(pgd_t) << (32 - PGDIR_SHIFT))
> +/* Bits to mask out from a PGD to get to the PUD page */
> +#define PGD_MASKED_BITS 0
>
> #define pte_ERROR(e) \
> pr_err("%s:%d: bad pte %llx.\n", __FILE__, __LINE__, \
> @@ -283,15 +295,6 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
> #define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 3 })
> #define __swp_entry_to_pte(x) ((pte_t) { (x).val << 3 })
>
> -#ifndef CONFIG_PPC_4K_PAGES
> -void pgtable_cache_init(void);
> -#else
> -/*
> - * No page table caches to initialise
> - */
> -#define pgtable_cache_init() do { } while (0)
> -#endif
> -
> extern int get_pteptr(struct mm_struct *mm, unsigned long addr, pte_t **ptep,
> pmd_t **pmdp);
>
> diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
> index 9fd77f8..0a46a5f 100644
> --- a/arch/powerpc/include/asm/book3s/64/pgtable.h
> +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
> @@ -789,9 +789,6 @@ extern struct page *pgd_page(pgd_t pgd);
> #define pgd_ERROR(e) \
> pr_err("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
>
> -void pgtable_cache_add(unsigned shift, void (*ctor)(void *));
> -void pgtable_cache_init(void);
> -
> static inline int map_kernel_page(unsigned long ea, unsigned long pa,
> unsigned long flags)
> {
> diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/include/asm/nohash/32/pgalloc.h
> index 76d6b9e..6331392 100644
> --- a/arch/powerpc/include/asm/nohash/32/pgalloc.h
> +++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h
> @@ -2,14 +2,42 @@
> #define _ASM_POWERPC_PGALLOC_32_H
>
> #include <linux/threads.h>
> +#include <linux/slab.h>
>
> -/* For 32-bit, all levels of page tables are just drawn from get_free_page() */
> -#define MAX_PGTABLE_INDEX_SIZE 0
> +/*
> + * Functions that deal with pagetables that could be at any level of
> + * the table need to be passed an "index_size" so they know how to
> + * handle allocation. For PTE pages (which are linked to a struct
> + * page for now, and drawn from the main get_free_pages() pool), the
> + * allocation size will be (2^index_size * sizeof(pointer)) and
> + * allocations are drawn from the kmem_cache in PGT_CACHE(index_size).
> + *
> + * The maximum index size needs to be big enough to allow any
> + * pagetable sizes we need, but small enough to fit in the low bits of
> + * any page table pointer. In other words all pagetables, even tiny
> + * ones, must be aligned to allow at least enough low 0 bits to
> + * contain this value. This value is also used as a mask, so it must
> + * be one less than a power of two.
> + */
> +#define MAX_PGTABLE_INDEX_SIZE 0xf
>
> extern void __bad_pte(pmd_t *pmd);
>
> -extern pgd_t *pgd_alloc(struct mm_struct *mm);
> -extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
> +extern struct kmem_cache *pgtable_cache[];
> +#define PGT_CACHE(shift) ({ \
> + BUG_ON(!(shift)); \
> + pgtable_cache[(shift) - 1]; \
> + })
> +
> +static inline pgd_t *pgd_alloc(struct mm_struct *mm)
> +{
> + return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL);
> +}
> +
> +static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
> +{
> + kmem_cache_free(PGT_CACHE(PGD_INDEX_SIZE), pgd);
> +}
>
> /*
> * We don't have any real pmd's, and this code never triggers because
> @@ -68,8 +96,12 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage)
>
> static inline void pgtable_free(void *table, unsigned index_size)
> {
> - BUG_ON(index_size); /* 32-bit doesn't use this */
> - free_page((unsigned long)table);
> + if (!index_size) {
> + free_page((unsigned long)table);
> + } else {
> + BUG_ON(index_size > MAX_PGTABLE_INDEX_SIZE);
> + kmem_cache_free(PGT_CACHE(index_size), table);
> + }
> }
>
> #define check_pgt_cache() do { } while (0)
> diff --git a/arch/powerpc/include/asm/nohash/32/pgtable.h b/arch/powerpc/include/asm/nohash/32/pgtable.h
> index c219ef7..8cbe222 100644
> --- a/arch/powerpc/include/asm/nohash/32/pgtable.h
> +++ b/arch/powerpc/include/asm/nohash/32/pgtable.h
> @@ -16,6 +16,26 @@ extern int icache_44x_need_flush;
>
> #endif /* __ASSEMBLY__ */
>
> +#define PTE_INDEX_SIZE PTE_SHIFT
> +#define PMD_INDEX_SIZE 0
> +#define PUD_INDEX_SIZE 0
> +#define PGD_INDEX_SIZE (32 - PGDIR_SHIFT)
> +
> +#define PMD_CACHE_INDEX PMD_INDEX_SIZE
> +
> +#ifndef __ASSEMBLY__
> +#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE)
> +#define PMD_TABLE_SIZE (sizeof(pmd_t) << PTE_INDEX_SIZE)
> +#define PUD_TABLE_SIZE (sizeof(pud_t) << PTE_INDEX_SIZE)
> +#define PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE)
> +#endif /* __ASSEMBLY__ */
> +
Same, please comment on why those TABLE sizes ?
> +#define PTRS_PER_PTE (1 << PTE_INDEX_SIZE)
> +#define PTRS_PER_PGD (1 << PGD_INDEX_SIZE)
> +
> +/* With 4k base page size, hugepage PTEs go at the PMD level */
> +#define MIN_HUGEPTE_SHIFT PMD_SHIFT
> +
> /*
> * The normal case is that PTEs are 32-bits and we have a 1-page
> * 1024-entry pgdir pointing to 1-page 1024-entry PTE pages. -- paulus
> @@ -27,22 +47,12 @@ extern int icache_44x_need_flush;
> * -Matt
> */
> /* PGDIR_SHIFT determines what a top-level page table entry can map */
> -#define PGDIR_SHIFT (PAGE_SHIFT + PTE_SHIFT)
> +#define PGDIR_SHIFT (PAGE_SHIFT + PTE_INDEX_SIZE)
> #define PGDIR_SIZE (1UL << PGDIR_SHIFT)
> #define PGDIR_MASK (~(PGDIR_SIZE-1))
>
> -/*
> - * entries per page directory level: our page-table tree is two-level, so
> - * we don't really have any PMD directory.
> - */
> -#ifndef __ASSEMBLY__
> -#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_SHIFT)
> -#define PGD_TABLE_SIZE (sizeof(pgd_t) << (32 - PGDIR_SHIFT))
> -#endif /* __ASSEMBLY__ */
> -
> -#define PTRS_PER_PTE (1 << PTE_SHIFT)
> -#define PTRS_PER_PMD 1
> -#define PTRS_PER_PGD (1 << (32 - PGDIR_SHIFT))
> +/* Bits to mask out from a PGD to get to the PUD page */
> +#define PGD_MASKED_BITS 0
>
> #define USER_PTRS_PER_PGD (TASK_SIZE / PGDIR_SIZE)
> #define FIRST_USER_ADDRESS 0UL
> @@ -328,15 +338,6 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
> #define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 3 })
> #define __swp_entry_to_pte(x) ((pte_t) { (x).val << 3 })
>
> -#ifndef CONFIG_PPC_4K_PAGES
> -void pgtable_cache_init(void);
> -#else
> -/*
> - * No page table caches to initialise
> - */
> -#define pgtable_cache_init() do { } while (0)
> -#endif
> -
> extern int get_pteptr(struct mm_struct *mm, unsigned long addr, pte_t **ptep,
> pmd_t **pmdp);
>
> diff --git a/arch/powerpc/include/asm/nohash/64/pgtable.h b/arch/powerpc/include/asm/nohash/64/pgtable.h
> index 653a183..619018a 100644
> --- a/arch/powerpc/include/asm/nohash/64/pgtable.h
> +++ b/arch/powerpc/include/asm/nohash/64/pgtable.h
> @@ -358,8 +358,6 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
> #define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val((pte)) })
> #define __swp_entry_to_pte(x) __pte((x).val)
>
> -void pgtable_cache_add(unsigned shift, void (*ctor)(void *));
> -void pgtable_cache_init(void);
> extern int map_kernel_page(unsigned long ea, unsigned long pa,
> unsigned long flags);
> extern int __meminit vmemmap_create_mapping(unsigned long start,
> diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h
> index 9bd87f2..dd01212 100644
> --- a/arch/powerpc/include/asm/pgtable.h
> +++ b/arch/powerpc/include/asm/pgtable.h
> @@ -78,6 +78,8 @@ static inline pte_t *find_linux_pte_or_hugepte(pgd_t *pgdir, unsigned long ea,
>
> unsigned long vmalloc_to_phys(void *vmalloc_addr);
>
> +void pgtable_cache_add(unsigned shift, void (*ctor)(void *));
> +void pgtable_cache_init(void);
> #endif /* __ASSEMBLY__ */
>
> #endif /* _ASM_POWERPC_PGTABLE_H */
> diff --git a/arch/powerpc/mm/Makefile b/arch/powerpc/mm/Makefile
> index 1a4e570..e8a86d2 100644
> --- a/arch/powerpc/mm/Makefile
> +++ b/arch/powerpc/mm/Makefile
> @@ -7,7 +7,8 @@ subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
> ccflags-$(CONFIG_PPC64) := $(NO_MINIMAL_TOC)
>
> obj-y := fault.o mem.o pgtable.o mmap.o \
> - init_$(BITS).o pgtable_$(BITS).o
> + init_$(BITS).o pgtable_$(BITS).o \
> + init-common.o
> obj-$(CONFIG_PPC_MMU_NOHASH) += mmu_context_nohash.o tlb_nohash.o \
> tlb_nohash_low.o
> obj-$(CONFIG_PPC_BOOK3E) += tlb_low_$(BITS)e.o
> diff --git a/arch/powerpc/mm/init-common.c b/arch/powerpc/mm/init-common.c
> new file mode 100644
> index 0000000..ab2b947
> --- /dev/null
> +++ b/arch/powerpc/mm/init-common.c
> @@ -0,0 +1,147 @@
> +/*
> + * PowerPC version
> + * Copyright (C) 1995-1996 Gary Thomas (gdt@linuxppc.org)
> + *
> + * Modifications by Paul Mackerras (PowerMac) (paulus@cs.anu.edu.au)
> + * and Cort Dougan (PReP) (cort@cs.nmt.edu)
> + * Copyright (C) 1996 Paul Mackerras
> + *
> + * Derived from "arch/i386/mm/init.c"
> + * Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
> + *
> + * Dave Engebretsen <engebret@us.ibm.com>
> + * Rework for PPC64 port.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + *
> + */
> +
> +#undef DEBUG
> +
> +#include <linux/signal.h>
> +#include <linux/sched.h>
> +#include <linux/kernel.h>
> +#include <linux/errno.h>
> +#include <linux/string.h>
> +#include <linux/types.h>
> +#include <linux/mman.h>
> +#include <linux/mm.h>
> +#include <linux/swap.h>
> +#include <linux/stddef.h>
> +#include <linux/vmalloc.h>
> +#include <linux/init.h>
> +#include <linux/delay.h>
> +#include <linux/highmem.h>
> +#include <linux/idr.h>
> +#include <linux/nodemask.h>
> +#include <linux/module.h>
> +#include <linux/poison.h>
> +#include <linux/memblock.h>
> +#include <linux/hugetlb.h>
> +#include <linux/slab.h>
> +
> +#include <asm/pgalloc.h>
> +#include <asm/page.h>
> +#include <asm/prom.h>
> +#include <asm/rtas.h>
> +#include <asm/io.h>
> +#include <asm/mmu_context.h>
> +#include <asm/pgtable.h>
> +#include <asm/mmu.h>
> +#include <asm/uaccess.h>
> +#include <asm/smp.h>
> +#include <asm/machdep.h>
> +#include <asm/tlb.h>
> +#include <asm/eeh.h>
> +#include <asm/processor.h>
> +#include <asm/mmzone.h>
> +#include <asm/cputable.h>
> +#include <asm/sections.h>
> +#include <asm/iommu.h>
> +#include <asm/vdso.h>
> +
> +#include "mmu_decl.h"
Do you need all these headers to get it compiled ?
> +
> +static void pgd_ctor(void *addr)
> +{
> + memset(addr, 0, PGD_TABLE_SIZE);
> +}
> +
> +static void pud_ctor(void *addr)
> +{
> + memset(addr, 0, PUD_TABLE_SIZE);
> +}
> +
> +static void pmd_ctor(void *addr)
> +{
> + memset(addr, 0, PMD_TABLE_SIZE);
> +}
> +
> +struct kmem_cache *pgtable_cache[MAX_PGTABLE_INDEX_SIZE];
> +
> +/*
> + * Create a kmem_cache() for pagetables. This is not used for PTE
> + * pages - they're linked to struct page, come from the normal free
> + * pages pool and have a different entry size (see real_pte_t) to
> + * everything else. Caches created by this function are used for all
> + * the higher level pagetables, and for hugepage pagetables.
> + */
> +void pgtable_cache_add(unsigned shift, void (*ctor)(void *))
> +{
> + char *name;
> + unsigned long table_size = sizeof(void *) << shift;
> + unsigned long align = table_size;
> +
> + /* When batching pgtable pointers for RCU freeing, we store
> + * the index size in the low bits. Table alignment must be
> + * big enough to fit it.
> + *
> + * Likewise, hugeapge pagetable pointers contain a (different)
> + * shift value in the low bits. All tables must be aligned so
> + * as to leave enough 0 bits in the address to contain it. */
> + unsigned long minalign = max(MAX_PGTABLE_INDEX_SIZE + 1,
> + HUGEPD_SHIFT_MASK + 1);
> + struct kmem_cache *new;
> +
> + /* It would be nice if this was a BUILD_BUG_ON(), but at the
> + * moment, gcc doesn't seem to recognize is_power_of_2 as a
> + * constant expression, so so much for that. */
> + BUG_ON(!is_power_of_2(minalign));
> + BUG_ON((shift < 1) || (shift > MAX_PGTABLE_INDEX_SIZE));
> +
> + if (PGT_CACHE(shift))
> + return; /* Already have a cache of this size */
> +
> + align = max_t(unsigned long, align, minalign);
> + name = kasprintf(GFP_KERNEL, "pgtable-2^%d", shift);
> + new = kmem_cache_create(name, table_size, align, 0, ctor);
> + kfree(name);
> + pgtable_cache[shift - 1] = new;
> + pr_debug("Allocated pgtable cache for order %d\n", shift);
> +}
> +
> +
> +void pgtable_cache_init(void)
> +{
> + pgtable_cache_add(PGD_INDEX_SIZE, pgd_ctor);
> +
> + if (PMD_INDEX_SIZE && !PGT_CACHE(PMD_INDEX_SIZE))
> + pgtable_cache_add(PMD_CACHE_INDEX, pmd_ctor);
> + /*
> + * In all current configs, when the PUD index exists it's the
> + * same size as either the pgd or pmd index except with THP enabled
> + * on book3s 64
> + */
> + if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
> + pgtable_cache_add(PUD_INDEX_SIZE, pud_ctor);
> +
> + if (!PGT_CACHE(PGD_INDEX_SIZE))
> + panic("Couldn't allocate pgd cache");
> + if (PMD_INDEX_SIZE && !PGT_CACHE(PMD_INDEX_SIZE))
> + panic("Couldn't allocate pmd pgtable caches");
> + if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
> + panic("Couldn't allocate pud pgtable caches");
> +}
> diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
> index 16ada1e..a000c35 100644
> --- a/arch/powerpc/mm/init_64.c
> +++ b/arch/powerpc/mm/init_64.c
> @@ -80,83 +80,6 @@ EXPORT_SYMBOL_GPL(memstart_addr);
> phys_addr_t kernstart_addr;
> EXPORT_SYMBOL_GPL(kernstart_addr);
>
> -static void pgd_ctor(void *addr)
> -{
> - memset(addr, 0, PGD_TABLE_SIZE);
> -}
> -
> -static void pud_ctor(void *addr)
> -{
> - memset(addr, 0, PUD_TABLE_SIZE);
> -}
> -
> -static void pmd_ctor(void *addr)
> -{
> - memset(addr, 0, PMD_TABLE_SIZE);
> -}
> -
> -struct kmem_cache *pgtable_cache[MAX_PGTABLE_INDEX_SIZE];
> -
> -/*
> - * Create a kmem_cache() for pagetables. This is not used for PTE
> - * pages - they're linked to struct page, come from the normal free
> - * pages pool and have a different entry size (see real_pte_t) to
> - * everything else. Caches created by this function are used for all
> - * the higher level pagetables, and for hugepage pagetables.
> - */
> -void pgtable_cache_add(unsigned shift, void (*ctor)(void *))
> -{
> - char *name;
> - unsigned long table_size = sizeof(void *) << shift;
> - unsigned long align = table_size;
> -
> - /* When batching pgtable pointers for RCU freeing, we store
> - * the index size in the low bits. Table alignment must be
> - * big enough to fit it.
> - *
> - * Likewise, hugeapge pagetable pointers contain a (different)
> - * shift value in the low bits. All tables must be aligned so
> - * as to leave enough 0 bits in the address to contain it. */
> - unsigned long minalign = max(MAX_PGTABLE_INDEX_SIZE + 1,
> - HUGEPD_SHIFT_MASK + 1);
> - struct kmem_cache *new;
> -
> - /* It would be nice if this was a BUILD_BUG_ON(), but at the
> - * moment, gcc doesn't seem to recognize is_power_of_2 as a
> - * constant expression, so so much for that. */
> - BUG_ON(!is_power_of_2(minalign));
> - BUG_ON((shift < 1) || (shift > MAX_PGTABLE_INDEX_SIZE));
> -
> - if (PGT_CACHE(shift))
> - return; /* Already have a cache of this size */
> -
> - align = max_t(unsigned long, align, minalign);
> - name = kasprintf(GFP_KERNEL, "pgtable-2^%d", shift);
> - new = kmem_cache_create(name, table_size, align, 0, ctor);
> - kfree(name);
> - pgtable_cache[shift - 1] = new;
> - pr_debug("Allocated pgtable cache for order %d\n", shift);
> -}
> -
> -
> -void pgtable_cache_init(void)
> -{
> - pgtable_cache_add(PGD_INDEX_SIZE, pgd_ctor);
> - pgtable_cache_add(PMD_CACHE_INDEX, pmd_ctor);
> - /*
> - * In all current configs, when the PUD index exists it's the
> - * same size as either the pgd or pmd index except with THP enabled
> - * on book3s 64
> - */
> - if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
> - pgtable_cache_add(PUD_INDEX_SIZE, pud_ctor);
> -
> - if (!PGT_CACHE(PGD_INDEX_SIZE) || !PGT_CACHE(PMD_CACHE_INDEX))
> - panic("Couldn't allocate pgtable caches");
> - if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
> - panic("Couldn't allocate pud pgtable caches");
> -}
> -
> #ifdef CONFIG_SPARSEMEM_VMEMMAP
> /*
> * Given an address within the vmemmap, determine the pfn of the page that
> diff --git a/arch/powerpc/mm/pgtable_32.c b/arch/powerpc/mm/pgtable_32.c
> index 0ae0572..a65c0b4 100644
> --- a/arch/powerpc/mm/pgtable_32.c
> +++ b/arch/powerpc/mm/pgtable_32.c
> @@ -42,43 +42,6 @@ EXPORT_SYMBOL(ioremap_bot); /* aka VMALLOC_END */
>
> extern char etext[], _stext[], _sinittext[], _einittext[];
>
> -#define PGDIR_ORDER (32 + PGD_T_LOG2 - PGDIR_SHIFT)
> -
> -#ifndef CONFIG_PPC_4K_PAGES
> -static struct kmem_cache *pgtable_cache;
> -
> -void pgtable_cache_init(void)
> -{
> - pgtable_cache = kmem_cache_create("PGDIR cache", 1 << PGDIR_ORDER,
> - 1 << PGDIR_ORDER, 0, NULL);
> - if (pgtable_cache == NULL)
> - panic("Couldn't allocate pgtable caches");
> -}
> -#endif
> -
> -pgd_t *pgd_alloc(struct mm_struct *mm)
> -{
> - pgd_t *ret;
> -
> - /* pgdir take page or two with 4K pages and a page fraction otherwise */
> -#ifndef CONFIG_PPC_4K_PAGES
> - ret = kmem_cache_alloc(pgtable_cache, GFP_KERNEL | __GFP_ZERO);
> -#else
> - ret = (pgd_t *)__get_free_pages(GFP_KERNEL|__GFP_ZERO,
> - PGDIR_ORDER - PAGE_SHIFT);
> -#endif
> - return ret;
> -}
> -
> -void pgd_free(struct mm_struct *mm, pgd_t *pgd)
> -{
> -#ifndef CONFIG_PPC_4K_PAGES
> - kmem_cache_free(pgtable_cache, (void *)pgd);
> -#else
> - free_pages((unsigned long)pgd, PGDIR_ORDER - PAGE_SHIFT);
> -#endif
> -}
> -
> __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
> {
> pte_t *pte;
> --
> 2.1.0
[toc] | [prev] | [next] | [standalone]
| From | christophe leroy <christophe.leroy@c-s.fr> |
|---|---|
| Date | 2016-09-19 20:50 +0200 |
| Subject | Re: [PATCH v2 1/3] powerpc: port 64 bits pgtable_cache to 32 bits |
| Message-ID | <sjbND-Bs-7@gated-at.bofh.it> |
| In reply to | #1486142 |
Le 19/09/2016 à 07:22, Aneesh Kumar K.V a écrit :
> Christophe Leroy <christophe.leroy@c-s.fr> writes:
>
>> Today powerpc64 uses a set of pgtable_caches while powerpc32 uses
>> standard pages when using 4k pages and a single pgtable_cache
>> if using other size pages.
>>
>> In preparation of implementing huge pages on the 8xx, this patch
>> replaces the specific powerpc32 handling by the 64 bits approach.
>>
>> This is done by:
>> * moving 64 bits pgtable_cache_add() and pgtable_cache_init()
>> in a new file called init-common.c
>> * modifying pgtable_cache_init() to also handle the case
>> without PMD
>> * removing the 32 bits version of pgtable_cache_add() and
>> pgtable_cache_init()
>> * copying related header contents from 64 bits into both the
>> book3s/32 and nohash/32 header files
>>
>> On the 8xx, the following cache sizes will be used:
>> * 4k pages mode:
>> - PGT_CACHE(10) for PGD
>> - PGT_CACHE(3) for 512k hugepage tables
>> * 16k pages mode:
>> - PGT_CACHE(6) for PGD
>> - PGT_CACHE(7) for 512k hugepage tables
>> - PGT_CACHE(3) for 8M hugepage tables
>>
>> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
>> ---
>> v2: in v1, hugepte_cache was wrongly replaced by PGT_CACHE(1).
>> This modification has been removed from v2.
>>
>> arch/powerpc/include/asm/book3s/32/pgalloc.h | 44 ++++++--
>> arch/powerpc/include/asm/book3s/32/pgtable.h | 43 ++++----
>> arch/powerpc/include/asm/book3s/64/pgtable.h | 3 -
>> arch/powerpc/include/asm/nohash/32/pgalloc.h | 44 ++++++--
>> arch/powerpc/include/asm/nohash/32/pgtable.h | 45 ++++----
>> arch/powerpc/include/asm/nohash/64/pgtable.h | 2 -
>> arch/powerpc/include/asm/pgtable.h | 2 +
>> arch/powerpc/mm/Makefile | 3 +-
>> arch/powerpc/mm/init-common.c | 147 +++++++++++++++++++++++++++
>> arch/powerpc/mm/init_64.c | 77 --------------
>> arch/powerpc/mm/pgtable_32.c | 37 -------
>> 11 files changed, 273 insertions(+), 174 deletions(-)
>> create mode 100644 arch/powerpc/mm/init-common.c
>>
>> diff --git a/arch/powerpc/include/asm/book3s/32/pgalloc.h b/arch/powerpc/include/asm/book3s/32/pgalloc.h
>> index 8e21bb4..d310546 100644
>> --- a/arch/powerpc/include/asm/book3s/32/pgalloc.h
>> +++ b/arch/powerpc/include/asm/book3s/32/pgalloc.h
>> @@ -2,14 +2,42 @@
>> #define _ASM_POWERPC_BOOK3S_32_PGALLOC_H
>>
>> #include <linux/threads.h>
>> +#include <linux/slab.h>
>>
>> -/* For 32-bit, all levels of page tables are just drawn from get_free_page() */
>> -#define MAX_PGTABLE_INDEX_SIZE 0
>> +/*
>> + * Functions that deal with pagetables that could be at any level of
>> + * the table need to be passed an "index_size" so they know how to
>> + * handle allocation. For PTE pages (which are linked to a struct
>> + * page for now, and drawn from the main get_free_pages() pool), the
>> + * allocation size will be (2^index_size * sizeof(pointer)) and
>> + * allocations are drawn from the kmem_cache in PGT_CACHE(index_size).
>> + *
>> + * The maximum index size needs to be big enough to allow any
>> + * pagetable sizes we need, but small enough to fit in the low bits of
>> + * any page table pointer. In other words all pagetables, even tiny
>> + * ones, must be aligned to allow at least enough low 0 bits to
>> + * contain this value. This value is also used as a mask, so it must
>> + * be one less than a power of two.
>> + */
>> +#define MAX_PGTABLE_INDEX_SIZE 0xf
>>
>> extern void __bad_pte(pmd_t *pmd);
>>
>> -extern pgd_t *pgd_alloc(struct mm_struct *mm);
>> -extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
>> +extern struct kmem_cache *pgtable_cache[];
>> +#define PGT_CACHE(shift) ({ \
>> + BUG_ON(!(shift)); \
>> + pgtable_cache[(shift) - 1]; \
>> + })
>> +
>> +static inline pgd_t *pgd_alloc(struct mm_struct *mm)
>> +{
>> + return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL);
>> +}
>> +
>> +static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
>> +{
>> + kmem_cache_free(PGT_CACHE(PGD_INDEX_SIZE), pgd);
>> +}
>>
>> /*
>> * We don't have any real pmd's, and this code never triggers because
>> @@ -68,8 +96,12 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage)
>>
>> static inline void pgtable_free(void *table, unsigned index_size)
>> {
>> - BUG_ON(index_size); /* 32-bit doesn't use this */
>> - free_page((unsigned long)table);
>> + if (!index_size) {
>> + free_page((unsigned long)table);
>> + } else {
>> + BUG_ON(index_size > MAX_PGTABLE_INDEX_SIZE);
>> + kmem_cache_free(PGT_CACHE(index_size), table);
>> + }
>> }
>>
>> #define check_pgt_cache() do { } while (0)
>> diff --git a/arch/powerpc/include/asm/book3s/32/pgtable.h b/arch/powerpc/include/asm/book3s/32/pgtable.h
>> index 6b8b2d5..f887499 100644
>> --- a/arch/powerpc/include/asm/book3s/32/pgtable.h
>> +++ b/arch/powerpc/include/asm/book3s/32/pgtable.h
>> @@ -8,6 +8,26 @@
>> /* And here we include common definitions */
>> #include <asm/pte-common.h>
>>
>> +#define PTE_INDEX_SIZE PTE_SHIFT
>> +#define PMD_INDEX_SIZE 0
>> +#define PUD_INDEX_SIZE 0
>> +#define PGD_INDEX_SIZE (32 - PGDIR_SHIFT)
>> +
>> +#define PMD_CACHE_INDEX PMD_INDEX_SIZE
>> +
>> +#ifndef __ASSEMBLY__
>> +#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE)
>> +#define PMD_TABLE_SIZE (sizeof(pmd_t) << PTE_INDEX_SIZE)
>> +#define PUD_TABLE_SIZE (sizeof(pud_t) << PTE_INDEX_SIZE)
>> +#define PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE)
>> +#endif /* __ASSEMBLY__ */
>
> Are these table size correct ? IIUC, We will have only PGD and PTE
> tables right ?
Oops, copy/paste error.
We won't have those tables, they won't be created as PMD_INDEX_SIZE and
PUD_INDEX_SIZE are 0. But they need to be defined to avoid compilation
errors of pmd_ctor() and pud_ctor()
Then what should I do ? Define them as 0, or just keep the standard
definition which follows ?
#define PMD_TABLE_SIZE (sizeof(pmd_t) << PMD_INDEX_SIZE)
#define PUD_TABLE_SIZE (sizeof(pud_t) << PUD_INDEX_SIZE)
>
>
>> +
>> +#define PTRS_PER_PTE (1 << PTE_INDEX_SIZE)
>> +#define PTRS_PER_PGD (1 << PGD_INDEX_SIZE)
>> +
>> +/* With 4k base page size, hugepage PTEs go at the PMD level */
>> +#define MIN_HUGEPTE_SHIFT PMD_SHIFT
>> +
>
> What does that comment mean ? I guess that came from copy-paste from
> other headers. I am not sure what it means there either other than the
> 64k hash config, where we place hugepage PTE at the PMD level. (ie, no hugepd).
Well, I'll remove the comment from here.
>
>
>> /*
>> * The normal case is that PTEs are 32-bits and we have a 1-page
>> * 1024-entry pgdir pointing to 1-page 1024-entry PTE pages. -- paulus
>> @@ -19,14 +39,10 @@
>> * -Matt
>> */
>> /* PGDIR_SHIFT determines what a top-level page table entry can map */
>> -#define PGDIR_SHIFT (PAGE_SHIFT + PTE_SHIFT)
>> +#define PGDIR_SHIFT (PAGE_SHIFT + PTE_INDEX_SIZE)
>> #define PGDIR_SIZE (1UL << PGDIR_SHIFT)
>> #define PGDIR_MASK (~(PGDIR_SIZE-1))
>>
>> -#define PTRS_PER_PTE (1 << PTE_SHIFT)
>> -#define PTRS_PER_PMD 1
>> -#define PTRS_PER_PGD (1 << (32 - PGDIR_SHIFT))
>> -
>> #define USER_PTRS_PER_PGD (TASK_SIZE / PGDIR_SIZE)
>> /*
>> * This is the bottom of the PKMAP area with HIGHMEM or an arbitrary
>> @@ -82,12 +98,8 @@
>>
>> extern unsigned long ioremap_bot;
>>
>> -/*
>> - * entries per page directory level: our page-table tree is two-level, so
>> - * we don't really have any PMD directory.
>> - */
>> -#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_SHIFT)
>> -#define PGD_TABLE_SIZE (sizeof(pgd_t) << (32 - PGDIR_SHIFT))
>> +/* Bits to mask out from a PGD to get to the PUD page */
>> +#define PGD_MASKED_BITS 0
>>
>> #define pte_ERROR(e) \
>> pr_err("%s:%d: bad pte %llx.\n", __FILE__, __LINE__, \
>> @@ -283,15 +295,6 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
>> #define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 3 })
>> #define __swp_entry_to_pte(x) ((pte_t) { (x).val << 3 })
>>
>> -#ifndef CONFIG_PPC_4K_PAGES
>> -void pgtable_cache_init(void);
>> -#else
>> -/*
>> - * No page table caches to initialise
>> - */
>> -#define pgtable_cache_init() do { } while (0)
>> -#endif
>> -
>> extern int get_pteptr(struct mm_struct *mm, unsigned long addr, pte_t **ptep,
>> pmd_t **pmdp);
>>
>> diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
>> index 9fd77f8..0a46a5f 100644
>> --- a/arch/powerpc/include/asm/book3s/64/pgtable.h
>> +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
>> @@ -789,9 +789,6 @@ extern struct page *pgd_page(pgd_t pgd);
>> #define pgd_ERROR(e) \
>> pr_err("%s:%d: bad pgd %08lx.\n", __FILE__, __LINE__, pgd_val(e))
>>
>> -void pgtable_cache_add(unsigned shift, void (*ctor)(void *));
>> -void pgtable_cache_init(void);
>> -
>> static inline int map_kernel_page(unsigned long ea, unsigned long pa,
>> unsigned long flags)
>> {
>> diff --git a/arch/powerpc/include/asm/nohash/32/pgalloc.h b/arch/powerpc/include/asm/nohash/32/pgalloc.h
>> index 76d6b9e..6331392 100644
>> --- a/arch/powerpc/include/asm/nohash/32/pgalloc.h
>> +++ b/arch/powerpc/include/asm/nohash/32/pgalloc.h
>> @@ -2,14 +2,42 @@
>> #define _ASM_POWERPC_PGALLOC_32_H
>>
>> #include <linux/threads.h>
>> +#include <linux/slab.h>
>>
>> -/* For 32-bit, all levels of page tables are just drawn from get_free_page() */
>> -#define MAX_PGTABLE_INDEX_SIZE 0
>> +/*
>> + * Functions that deal with pagetables that could be at any level of
>> + * the table need to be passed an "index_size" so they know how to
>> + * handle allocation. For PTE pages (which are linked to a struct
>> + * page for now, and drawn from the main get_free_pages() pool), the
>> + * allocation size will be (2^index_size * sizeof(pointer)) and
>> + * allocations are drawn from the kmem_cache in PGT_CACHE(index_size).
>> + *
>> + * The maximum index size needs to be big enough to allow any
>> + * pagetable sizes we need, but small enough to fit in the low bits of
>> + * any page table pointer. In other words all pagetables, even tiny
>> + * ones, must be aligned to allow at least enough low 0 bits to
>> + * contain this value. This value is also used as a mask, so it must
>> + * be one less than a power of two.
>> + */
>> +#define MAX_PGTABLE_INDEX_SIZE 0xf
>>
>> extern void __bad_pte(pmd_t *pmd);
>>
>> -extern pgd_t *pgd_alloc(struct mm_struct *mm);
>> -extern void pgd_free(struct mm_struct *mm, pgd_t *pgd);
>> +extern struct kmem_cache *pgtable_cache[];
>> +#define PGT_CACHE(shift) ({ \
>> + BUG_ON(!(shift)); \
>> + pgtable_cache[(shift) - 1]; \
>> + })
>> +
>> +static inline pgd_t *pgd_alloc(struct mm_struct *mm)
>> +{
>> + return kmem_cache_alloc(PGT_CACHE(PGD_INDEX_SIZE), GFP_KERNEL);
>> +}
>> +
>> +static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd)
>> +{
>> + kmem_cache_free(PGT_CACHE(PGD_INDEX_SIZE), pgd);
>> +}
>>
>> /*
>> * We don't have any real pmd's, and this code never triggers because
>> @@ -68,8 +96,12 @@ static inline void pte_free(struct mm_struct *mm, pgtable_t ptepage)
>>
>> static inline void pgtable_free(void *table, unsigned index_size)
>> {
>> - BUG_ON(index_size); /* 32-bit doesn't use this */
>> - free_page((unsigned long)table);
>> + if (!index_size) {
>> + free_page((unsigned long)table);
>> + } else {
>> + BUG_ON(index_size > MAX_PGTABLE_INDEX_SIZE);
>> + kmem_cache_free(PGT_CACHE(index_size), table);
>> + }
>> }
>>
>> #define check_pgt_cache() do { } while (0)
>> diff --git a/arch/powerpc/include/asm/nohash/32/pgtable.h b/arch/powerpc/include/asm/nohash/32/pgtable.h
>> index c219ef7..8cbe222 100644
>> --- a/arch/powerpc/include/asm/nohash/32/pgtable.h
>> +++ b/arch/powerpc/include/asm/nohash/32/pgtable.h
>> @@ -16,6 +16,26 @@ extern int icache_44x_need_flush;
>>
>> #endif /* __ASSEMBLY__ */
>>
>> +#define PTE_INDEX_SIZE PTE_SHIFT
>> +#define PMD_INDEX_SIZE 0
>> +#define PUD_INDEX_SIZE 0
>> +#define PGD_INDEX_SIZE (32 - PGDIR_SHIFT)
>> +
>> +#define PMD_CACHE_INDEX PMD_INDEX_SIZE
>> +
>> +#ifndef __ASSEMBLY__
>> +#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_INDEX_SIZE)
>> +#define PMD_TABLE_SIZE (sizeof(pmd_t) << PTE_INDEX_SIZE)
>> +#define PUD_TABLE_SIZE (sizeof(pud_t) << PTE_INDEX_SIZE)
>> +#define PGD_TABLE_SIZE (sizeof(pgd_t) << PGD_INDEX_SIZE)
>> +#endif /* __ASSEMBLY__ */
>> +
>
> Same, please comment on why those TABLE sizes ?
>
Yes same error, will fix it the same way.
>
>> +#define PTRS_PER_PTE (1 << PTE_INDEX_SIZE)
>> +#define PTRS_PER_PGD (1 << PGD_INDEX_SIZE)
>> +
>> +/* With 4k base page size, hugepage PTEs go at the PMD level */
>> +#define MIN_HUGEPTE_SHIFT PMD_SHIFT
>> +
>> /*
>> * The normal case is that PTEs are 32-bits and we have a 1-page
>> * 1024-entry pgdir pointing to 1-page 1024-entry PTE pages. -- paulus
>> @@ -27,22 +47,12 @@ extern int icache_44x_need_flush;
>> * -Matt
>> */
>> /* PGDIR_SHIFT determines what a top-level page table entry can map */
>> -#define PGDIR_SHIFT (PAGE_SHIFT + PTE_SHIFT)
>> +#define PGDIR_SHIFT (PAGE_SHIFT + PTE_INDEX_SIZE)
>> #define PGDIR_SIZE (1UL << PGDIR_SHIFT)
>> #define PGDIR_MASK (~(PGDIR_SIZE-1))
>>
>> -/*
>> - * entries per page directory level: our page-table tree is two-level, so
>> - * we don't really have any PMD directory.
>> - */
>> -#ifndef __ASSEMBLY__
>> -#define PTE_TABLE_SIZE (sizeof(pte_t) << PTE_SHIFT)
>> -#define PGD_TABLE_SIZE (sizeof(pgd_t) << (32 - PGDIR_SHIFT))
>> -#endif /* __ASSEMBLY__ */
>> -
>> -#define PTRS_PER_PTE (1 << PTE_SHIFT)
>> -#define PTRS_PER_PMD 1
>> -#define PTRS_PER_PGD (1 << (32 - PGDIR_SHIFT))
>> +/* Bits to mask out from a PGD to get to the PUD page */
>> +#define PGD_MASKED_BITS 0
>>
>> #define USER_PTRS_PER_PGD (TASK_SIZE / PGDIR_SIZE)
>> #define FIRST_USER_ADDRESS 0UL
>> @@ -328,15 +338,6 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
>> #define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) >> 3 })
>> #define __swp_entry_to_pte(x) ((pte_t) { (x).val << 3 })
>>
>> -#ifndef CONFIG_PPC_4K_PAGES
>> -void pgtable_cache_init(void);
>> -#else
>> -/*
>> - * No page table caches to initialise
>> - */
>> -#define pgtable_cache_init() do { } while (0)
>> -#endif
>> -
>> extern int get_pteptr(struct mm_struct *mm, unsigned long addr, pte_t **ptep,
>> pmd_t **pmdp);
>>
>> diff --git a/arch/powerpc/include/asm/nohash/64/pgtable.h b/arch/powerpc/include/asm/nohash/64/pgtable.h
>> index 653a183..619018a 100644
>> --- a/arch/powerpc/include/asm/nohash/64/pgtable.h
>> +++ b/arch/powerpc/include/asm/nohash/64/pgtable.h
>> @@ -358,8 +358,6 @@ static inline void __ptep_set_access_flags(struct mm_struct *mm,
>> #define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val((pte)) })
>> #define __swp_entry_to_pte(x) __pte((x).val)
>>
>> -void pgtable_cache_add(unsigned shift, void (*ctor)(void *));
>> -void pgtable_cache_init(void);
>> extern int map_kernel_page(unsigned long ea, unsigned long pa,
>> unsigned long flags);
>> extern int __meminit vmemmap_create_mapping(unsigned long start,
>> diff --git a/arch/powerpc/include/asm/pgtable.h b/arch/powerpc/include/asm/pgtable.h
>> index 9bd87f2..dd01212 100644
>> --- a/arch/powerpc/include/asm/pgtable.h
>> +++ b/arch/powerpc/include/asm/pgtable.h
>> @@ -78,6 +78,8 @@ static inline pte_t *find_linux_pte_or_hugepte(pgd_t *pgdir, unsigned long ea,
>>
>> unsigned long vmalloc_to_phys(void *vmalloc_addr);
>>
>> +void pgtable_cache_add(unsigned shift, void (*ctor)(void *));
>> +void pgtable_cache_init(void);
>> #endif /* __ASSEMBLY__ */
>>
>> #endif /* _ASM_POWERPC_PGTABLE_H */
>> diff --git a/arch/powerpc/mm/Makefile b/arch/powerpc/mm/Makefile
>> index 1a4e570..e8a86d2 100644
>> --- a/arch/powerpc/mm/Makefile
>> +++ b/arch/powerpc/mm/Makefile
>> @@ -7,7 +7,8 @@ subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
>> ccflags-$(CONFIG_PPC64) := $(NO_MINIMAL_TOC)
>>
>> obj-y := fault.o mem.o pgtable.o mmap.o \
>> - init_$(BITS).o pgtable_$(BITS).o
>> + init_$(BITS).o pgtable_$(BITS).o \
>> + init-common.o
>> obj-$(CONFIG_PPC_MMU_NOHASH) += mmu_context_nohash.o tlb_nohash.o \
>> tlb_nohash_low.o
>> obj-$(CONFIG_PPC_BOOK3E) += tlb_low_$(BITS)e.o
>> diff --git a/arch/powerpc/mm/init-common.c b/arch/powerpc/mm/init-common.c
>> new file mode 100644
>> index 0000000..ab2b947
>> --- /dev/null
>> +++ b/arch/powerpc/mm/init-common.c
>> @@ -0,0 +1,147 @@
>> +/*
>> + * PowerPC version
>> + * Copyright (C) 1995-1996 Gary Thomas (gdt@linuxppc.org)
>> + *
>> + * Modifications by Paul Mackerras (PowerMac) (paulus@cs.anu.edu.au)
>> + * and Cort Dougan (PReP) (cort@cs.nmt.edu)
>> + * Copyright (C) 1996 Paul Mackerras
>> + *
>> + * Derived from "arch/i386/mm/init.c"
>> + * Copyright (C) 1991, 1992, 1993, 1994 Linus Torvalds
>> + *
>> + * Dave Engebretsen <engebret@us.ibm.com>
>> + * Rework for PPC64 port.
>> + *
>> + * This program is free software; you can redistribute it and/or
>> + * modify it under the terms of the GNU General Public License
>> + * as published by the Free Software Foundation; either version
>> + * 2 of the License, or (at your option) any later version.
>> + *
>> + */
>> +
>> +#undef DEBUG
>> +
>> +#include <linux/signal.h>
>> +#include <linux/sched.h>
>> +#include <linux/kernel.h>
>> +#include <linux/errno.h>
>> +#include <linux/string.h>
>> +#include <linux/types.h>
>> +#include <linux/mman.h>
>> +#include <linux/mm.h>
>> +#include <linux/swap.h>
>> +#include <linux/stddef.h>
>> +#include <linux/vmalloc.h>
>> +#include <linux/init.h>
>> +#include <linux/delay.h>
>> +#include <linux/highmem.h>
>> +#include <linux/idr.h>
>> +#include <linux/nodemask.h>
>> +#include <linux/module.h>
>> +#include <linux/poison.h>
>> +#include <linux/memblock.h>
>> +#include <linux/hugetlb.h>
>> +#include <linux/slab.h>
>> +
>> +#include <asm/pgalloc.h>
>> +#include <asm/page.h>
>> +#include <asm/prom.h>
>> +#include <asm/rtas.h>
>> +#include <asm/io.h>
>> +#include <asm/mmu_context.h>
>> +#include <asm/pgtable.h>
>> +#include <asm/mmu.h>
>> +#include <asm/uaccess.h>
>> +#include <asm/smp.h>
>> +#include <asm/machdep.h>
>> +#include <asm/tlb.h>
>> +#include <asm/eeh.h>
>> +#include <asm/processor.h>
>> +#include <asm/mmzone.h>
>> +#include <asm/cputable.h>
>> +#include <asm/sections.h>
>> +#include <asm/iommu.h>
>> +#include <asm/vdso.h>
>> +
>> +#include "mmu_decl.h"
>
>
> Do you need all these headers to get it compiled ?
>
Probably not, I just copied the ones from init_64.c
Will clean it up.
>
>> +
>> +static void pgd_ctor(void *addr)
>> +{
>> + memset(addr, 0, PGD_TABLE_SIZE);
>> +}
>> +
>> +static void pud_ctor(void *addr)
>> +{
>> + memset(addr, 0, PUD_TABLE_SIZE);
>> +}
>> +
>> +static void pmd_ctor(void *addr)
>> +{
>> + memset(addr, 0, PMD_TABLE_SIZE);
>> +}
>> +
>> +struct kmem_cache *pgtable_cache[MAX_PGTABLE_INDEX_SIZE];
>> +
>> +/*
>> + * Create a kmem_cache() for pagetables. This is not used for PTE
>> + * pages - they're linked to struct page, come from the normal free
>> + * pages pool and have a different entry size (see real_pte_t) to
>> + * everything else. Caches created by this function are used for all
>> + * the higher level pagetables, and for hugepage pagetables.
>> + */
>> +void pgtable_cache_add(unsigned shift, void (*ctor)(void *))
>> +{
>> + char *name;
>> + unsigned long table_size = sizeof(void *) << shift;
>> + unsigned long align = table_size;
>> +
>> + /* When batching pgtable pointers for RCU freeing, we store
>> + * the index size in the low bits. Table alignment must be
>> + * big enough to fit it.
>> + *
>> + * Likewise, hugeapge pagetable pointers contain a (different)
>> + * shift value in the low bits. All tables must be aligned so
>> + * as to leave enough 0 bits in the address to contain it. */
>> + unsigned long minalign = max(MAX_PGTABLE_INDEX_SIZE + 1,
>> + HUGEPD_SHIFT_MASK + 1);
>> + struct kmem_cache *new;
>> +
>> + /* It would be nice if this was a BUILD_BUG_ON(), but at the
>> + * moment, gcc doesn't seem to recognize is_power_of_2 as a
>> + * constant expression, so so much for that. */
>> + BUG_ON(!is_power_of_2(minalign));
>> + BUG_ON((shift < 1) || (shift > MAX_PGTABLE_INDEX_SIZE));
>> +
>> + if (PGT_CACHE(shift))
>> + return; /* Already have a cache of this size */
>> +
>> + align = max_t(unsigned long, align, minalign);
>> + name = kasprintf(GFP_KERNEL, "pgtable-2^%d", shift);
>> + new = kmem_cache_create(name, table_size, align, 0, ctor);
>> + kfree(name);
>> + pgtable_cache[shift - 1] = new;
>> + pr_debug("Allocated pgtable cache for order %d\n", shift);
>> +}
>> +
>> +
>> +void pgtable_cache_init(void)
>> +{
>> + pgtable_cache_add(PGD_INDEX_SIZE, pgd_ctor);
>> +
>> + if (PMD_INDEX_SIZE && !PGT_CACHE(PMD_INDEX_SIZE))
>> + pgtable_cache_add(PMD_CACHE_INDEX, pmd_ctor);
>> + /*
>> + * In all current configs, when the PUD index exists it's the
>> + * same size as either the pgd or pmd index except with THP enabled
>> + * on book3s 64
>> + */
>> + if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
>> + pgtable_cache_add(PUD_INDEX_SIZE, pud_ctor);
>> +
>> + if (!PGT_CACHE(PGD_INDEX_SIZE))
>> + panic("Couldn't allocate pgd cache");
>> + if (PMD_INDEX_SIZE && !PGT_CACHE(PMD_INDEX_SIZE))
>> + panic("Couldn't allocate pmd pgtable caches");
>> + if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
>> + panic("Couldn't allocate pud pgtable caches");
>> +}
>> diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c
>> index 16ada1e..a000c35 100644
>> --- a/arch/powerpc/mm/init_64.c
>> +++ b/arch/powerpc/mm/init_64.c
>> @@ -80,83 +80,6 @@ EXPORT_SYMBOL_GPL(memstart_addr);
>> phys_addr_t kernstart_addr;
>> EXPORT_SYMBOL_GPL(kernstart_addr);
>>
>> -static void pgd_ctor(void *addr)
>> -{
>> - memset(addr, 0, PGD_TABLE_SIZE);
>> -}
>> -
>> -static void pud_ctor(void *addr)
>> -{
>> - memset(addr, 0, PUD_TABLE_SIZE);
>> -}
>> -
>> -static void pmd_ctor(void *addr)
>> -{
>> - memset(addr, 0, PMD_TABLE_SIZE);
>> -}
>> -
>> -struct kmem_cache *pgtable_cache[MAX_PGTABLE_INDEX_SIZE];
>> -
>> -/*
>> - * Create a kmem_cache() for pagetables. This is not used for PTE
>> - * pages - they're linked to struct page, come from the normal free
>> - * pages pool and have a different entry size (see real_pte_t) to
>> - * everything else. Caches created by this function are used for all
>> - * the higher level pagetables, and for hugepage pagetables.
>> - */
>> -void pgtable_cache_add(unsigned shift, void (*ctor)(void *))
>> -{
>> - char *name;
>> - unsigned long table_size = sizeof(void *) << shift;
>> - unsigned long align = table_size;
>> -
>> - /* When batching pgtable pointers for RCU freeing, we store
>> - * the index size in the low bits. Table alignment must be
>> - * big enough to fit it.
>> - *
>> - * Likewise, hugeapge pagetable pointers contain a (different)
>> - * shift value in the low bits. All tables must be aligned so
>> - * as to leave enough 0 bits in the address to contain it. */
>> - unsigned long minalign = max(MAX_PGTABLE_INDEX_SIZE + 1,
>> - HUGEPD_SHIFT_MASK + 1);
>> - struct kmem_cache *new;
>> -
>> - /* It would be nice if this was a BUILD_BUG_ON(), but at the
>> - * moment, gcc doesn't seem to recognize is_power_of_2 as a
>> - * constant expression, so so much for that. */
>> - BUG_ON(!is_power_of_2(minalign));
>> - BUG_ON((shift < 1) || (shift > MAX_PGTABLE_INDEX_SIZE));
>> -
>> - if (PGT_CACHE(shift))
>> - return; /* Already have a cache of this size */
>> -
>> - align = max_t(unsigned long, align, minalign);
>> - name = kasprintf(GFP_KERNEL, "pgtable-2^%d", shift);
>> - new = kmem_cache_create(name, table_size, align, 0, ctor);
>> - kfree(name);
>> - pgtable_cache[shift - 1] = new;
>> - pr_debug("Allocated pgtable cache for order %d\n", shift);
>> -}
>> -
>> -
>> -void pgtable_cache_init(void)
>> -{
>> - pgtable_cache_add(PGD_INDEX_SIZE, pgd_ctor);
>> - pgtable_cache_add(PMD_CACHE_INDEX, pmd_ctor);
>> - /*
>> - * In all current configs, when the PUD index exists it's the
>> - * same size as either the pgd or pmd index except with THP enabled
>> - * on book3s 64
>> - */
>> - if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
>> - pgtable_cache_add(PUD_INDEX_SIZE, pud_ctor);
>> -
>> - if (!PGT_CACHE(PGD_INDEX_SIZE) || !PGT_CACHE(PMD_CACHE_INDEX))
>> - panic("Couldn't allocate pgtable caches");
>> - if (PUD_INDEX_SIZE && !PGT_CACHE(PUD_INDEX_SIZE))
>> - panic("Couldn't allocate pud pgtable caches");
>> -}
>> -
>> #ifdef CONFIG_SPARSEMEM_VMEMMAP
>> /*
>> * Given an address within the vmemmap, determine the pfn of the page that
>> diff --git a/arch/powerpc/mm/pgtable_32.c b/arch/powerpc/mm/pgtable_32.c
>> index 0ae0572..a65c0b4 100644
>> --- a/arch/powerpc/mm/pgtable_32.c
>> +++ b/arch/powerpc/mm/pgtable_32.c
>> @@ -42,43 +42,6 @@ EXPORT_SYMBOL(ioremap_bot); /* aka VMALLOC_END */
>>
>> extern char etext[], _stext[], _sinittext[], _einittext[];
>>
>> -#define PGDIR_ORDER (32 + PGD_T_LOG2 - PGDIR_SHIFT)
>> -
>> -#ifndef CONFIG_PPC_4K_PAGES
>> -static struct kmem_cache *pgtable_cache;
>> -
>> -void pgtable_cache_init(void)
>> -{
>> - pgtable_cache = kmem_cache_create("PGDIR cache", 1 << PGDIR_ORDER,
>> - 1 << PGDIR_ORDER, 0, NULL);
>> - if (pgtable_cache == NULL)
>> - panic("Couldn't allocate pgtable caches");
>> -}
>> -#endif
>> -
>> -pgd_t *pgd_alloc(struct mm_struct *mm)
>> -{
>> - pgd_t *ret;
>> -
>> - /* pgdir take page or two with 4K pages and a page fraction otherwise */
>> -#ifndef CONFIG_PPC_4K_PAGES
>> - ret = kmem_cache_alloc(pgtable_cache, GFP_KERNEL | __GFP_ZERO);
>> -#else
>> - ret = (pgd_t *)__get_free_pages(GFP_KERNEL|__GFP_ZERO,
>> - PGDIR_ORDER - PAGE_SHIFT);
>> -#endif
>> - return ret;
>> -}
>> -
>> -void pgd_free(struct mm_struct *mm, pgd_t *pgd)
>> -{
>> -#ifndef CONFIG_PPC_4K_PAGES
>> - kmem_cache_free(pgtable_cache, (void *)pgd);
>> -#else
>> - free_pages((unsigned long)pgd, PGDIR_ORDER - PAGE_SHIFT);
>> -#endif
>> -}
>> -
>> __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm, unsigned long address)
>> {
>> pte_t *pte;
>> --
>> 2.1.0
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
[toc] | [prev] | [next] | [standalone]
| From | Christophe Leroy <christophe.leroy@c-s.fr> |
|---|---|
| Date | 2016-09-16 09:50 +0200 |
| Subject | [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic |
| Message-ID | <shW4h-1Eu-17@gated-at.bofh.it> |
| In reply to | #1484745 |
Today there are two implementations of hugetlbpages which are managed
by exclusive #ifdefs:
* FSL_BOOKE: several directory entries points to the same single hugepage
* BOOK3S: one upper level directory entry points to a table of hugepages
In preparation of implementation of hugepage support on the 8xx, we
need a mix of the two above solutions, because the 8xx needs both cases
depending on the size of pages:
* In 4k page size mode, each PGD entry covers a 4M bytes area. It means
that 2 PGD entries will be necessary to cover an 8M hugepage while a
single PGD entry will cover 8x 512k hugepages.
* In 16 page size mode, each PGD entry covers a 64M bytes area. It means
that 8x 8M hugepages will be covered by one PGD entry and 64x 512k
hugepages will be covers by one PGD entry.
This patch:
* removes #ifdefs in favor of if/else based on the range sizes
* merges the two huge_pte_alloc() functions as they are pretty similar
* merges the two hugetlbpage_init() functions as they are pretty similar
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
---
v2: This part is new and results from a split of last patch of v1 serie in
two parts
arch/powerpc/mm/hugetlbpage.c | 189 +++++++++++++++++-------------------------
1 file changed, 77 insertions(+), 112 deletions(-)
diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
index 8a512b1..2119f00 100644
--- a/arch/powerpc/mm/hugetlbpage.c
+++ b/arch/powerpc/mm/hugetlbpage.c
@@ -64,14 +64,16 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
{
struct kmem_cache *cachep;
pte_t *new;
-
-#ifdef CONFIG_PPC_FSL_BOOK3E
int i;
- int num_hugepd = 1 << (pshift - pdshift);
- cachep = hugepte_cache;
-#else
- cachep = PGT_CACHE(pdshift - pshift);
-#endif
+ int num_hugepd;
+
+ if (pshift >= pdshift) {
+ cachep = hugepte_cache;
+ num_hugepd = 1 << (pshift - pdshift);
+ } else {
+ cachep = PGT_CACHE(pdshift - pshift);
+ num_hugepd = 1;
+ }
new = kmem_cache_zalloc(cachep, GFP_KERNEL);
@@ -89,7 +91,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
smp_wmb();
spin_lock(&mm->page_table_lock);
-#ifdef CONFIG_PPC_FSL_BOOK3E
+
/*
* We have multiple higher-level entries that point to the same
* actual pte location. Fill in each as we go and backtrack on error.
@@ -100,8 +102,13 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
if (unlikely(!hugepd_none(*hpdp)))
break;
else
+#ifdef CONFIG_PPC_BOOK3S_64
+ hpdp->pd = __pa(new) |
+ (shift_to_mmu_psize(pshift) << 2);
+#else
/* We use the old format for PPC_FSL_BOOK3E */
hpdp->pd = ((unsigned long)new & ~PD_HUGE) | pshift;
+#endif
}
/* If we bailed from the for loop early, an error occurred, clean up */
if (i < num_hugepd) {
@@ -109,17 +116,6 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
hpdp->pd = 0;
kmem_cache_free(cachep, new);
}
-#else
- if (!hugepd_none(*hpdp))
- kmem_cache_free(cachep, new);
- else {
-#ifdef CONFIG_PPC_BOOK3S_64
- hpdp->pd = __pa(new) | (shift_to_mmu_psize(pshift) << 2);
-#else
- hpdp->pd = ((unsigned long)new & ~PD_HUGE) | pshift;
-#endif
- }
-#endif
spin_unlock(&mm->page_table_lock);
return 0;
}
@@ -136,7 +132,6 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
#define HUGEPD_PUD_SHIFT PMD_SHIFT
#endif
-#ifdef CONFIG_PPC_BOOK3S_64
/*
* At this point we do the placement change only for BOOK3S 64. This would
* possibly work on other subarchs.
@@ -153,6 +148,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz
addr &= ~(sz-1);
pg = pgd_offset(mm, addr);
+#ifdef CONFIG_PPC_BOOK3S_64
if (pshift == PGDIR_SHIFT)
/* 16GB huge page */
return (pte_t *) pg;
@@ -178,32 +174,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz
hpdp = (hugepd_t *)pm;
}
}
- if (!hpdp)
- return NULL;
-
- BUG_ON(!hugepd_none(*hpdp) && !hugepd_ok(*hpdp));
-
- if (hugepd_none(*hpdp) && __hugepte_alloc(mm, hpdp, addr, pdshift, pshift))
- return NULL;
-
- return hugepte_offset(*hpdp, addr, pdshift);
-}
-
#else
-
-pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz)
-{
- pgd_t *pg;
- pud_t *pu;
- pmd_t *pm;
- hugepd_t *hpdp = NULL;
- unsigned pshift = __ffs(sz);
- unsigned pdshift = PGDIR_SHIFT;
-
- addr &= ~(sz-1);
-
- pg = pgd_offset(mm, addr);
-
if (pshift >= HUGEPD_PGD_SHIFT) {
hpdp = (hugepd_t *)pg;
} else {
@@ -217,7 +188,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz
hpdp = (hugepd_t *)pm;
}
}
-
+#endif
if (!hpdp)
return NULL;
@@ -228,7 +199,6 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, unsigned long addr, unsigned long sz
return hugepte_offset(*hpdp, addr, pdshift);
}
-#endif
#ifdef CONFIG_PPC_FSL_BOOK3E
/* Build list of addresses of gigantic pages. This function is used in early
@@ -310,7 +280,11 @@ static int __init do_gpage_early_setup(char *param, char *val,
npages = 0;
if (npages > MAX_NUMBER_GPAGES) {
pr_warn("MMU: %lu pages requested for page "
+#ifdef CONFIG_PHYS_ADDR_T_64BIT
"size %llu KB, limiting to "
+#else
+ "size %u KB, limiting to "
+#endif
__stringify(MAX_NUMBER_GPAGES) "\n",
npages, size / 1024);
npages = MAX_NUMBER_GPAGES;
@@ -442,6 +416,12 @@ static void hugepd_free(struct mmu_gather *tlb, void *hugepte)
}
put_cpu_var(hugepd_freelist_cur);
}
+#else
+static void hugepd_free(struct mmu_gather *tlb, void *hugepte)
+{
+ BUG();
+}
+
#endif
static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshift,
@@ -453,13 +433,11 @@ static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshif
unsigned long pdmask = ~((1UL << pdshift) - 1);
unsigned int num_hugepd = 1;
+ unsigned int shift = hugepd_shift(*hpdp);
-#ifdef CONFIG_PPC_FSL_BOOK3E
/* Note: On fsl the hpdp may be the first of several */
- num_hugepd = (1 << (hugepd_shift(*hpdp) - pdshift));
-#else
- unsigned int shift = hugepd_shift(*hpdp);
-#endif
+ if (shift > pdshift)
+ num_hugepd = 1 << (shift - pdshift);
start &= pdmask;
if (start < floor)
@@ -475,11 +453,10 @@ static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshif
for (i = 0; i < num_hugepd; i++, hpdp++)
hpdp->pd = 0;
-#ifdef CONFIG_PPC_FSL_BOOK3E
- hugepd_free(tlb, hugepte);
-#else
- pgtable_free_tlb(tlb, hugepte, pdshift - shift);
-#endif
+ if (shift >= pdshift)
+ hugepd_free(tlb, hugepte);
+ else
+ pgtable_free_tlb(tlb, hugepte, pdshift - shift);
}
static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
@@ -492,6 +469,8 @@ static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
start = addr;
do {
+ unsigned long more;
+
pmd = pmd_offset(pud, addr);
next = pmd_addr_end(addr, end);
if (!is_hugepd(__hugepd(pmd_val(*pmd)))) {
@@ -502,15 +481,16 @@ static void hugetlb_free_pmd_range(struct mmu_gather *tlb, pud_t *pud,
WARN_ON(!pmd_none_or_clear_bad(pmd));
continue;
}
-#ifdef CONFIG_PPC_FSL_BOOK3E
/*
* Increment next by the size of the huge mapping since
* there may be more than one entry at this level for a
* single hugepage, but all of them point to
* the same kmem cache that holds the hugepte.
*/
- next = addr + (1 << hugepd_shift(*(hugepd_t *)pmd));
-#endif
+ more = addr + (1 << hugepd_shift(*(hugepd_t *)pmd));
+ if (more > next)
+ next = more;
+
free_hugepd_range(tlb, (hugepd_t *)pmd, PMD_SHIFT,
addr, next, floor, ceiling);
} while (addr = next, addr != end);
@@ -550,15 +530,17 @@ static void hugetlb_free_pud_range(struct mmu_gather *tlb, pgd_t *pgd,
hugetlb_free_pmd_range(tlb, pud, addr, next, floor,
ceiling);
} else {
-#ifdef CONFIG_PPC_FSL_BOOK3E
+ unsigned long more;
/*
* Increment next by the size of the huge mapping since
* there may be more than one entry at this level for a
* single hugepage, but all of them point to
* the same kmem cache that holds the hugepte.
*/
- next = addr + (1 << hugepd_shift(*(hugepd_t *)pud));
-#endif
+ more = addr + (1 << hugepd_shift(*(hugepd_t *)pud));
+ if (more > next)
+ next = more;
+
free_hugepd_range(tlb, (hugepd_t *)pud, PUD_SHIFT,
addr, next, floor, ceiling);
}
@@ -615,15 +597,17 @@ void hugetlb_free_pgd_range(struct mmu_gather *tlb,
continue;
hugetlb_free_pud_range(tlb, pgd, addr, next, floor, ceiling);
} else {
-#ifdef CONFIG_PPC_FSL_BOOK3E
+ unsigned long more;
/*
* Increment next by the size of the huge mapping since
* there may be more than one entry at the pgd level
* for a single hugepage, but all of them point to the
* same kmem cache that holds the hugepte.
*/
- next = addr + (1 << hugepd_shift(*(hugepd_t *)pgd));
-#endif
+ more = addr + (1 << hugepd_shift(*(hugepd_t *)pgd));
+ if (more > next)
+ next = more;
+
free_hugepd_range(tlb, (hugepd_t *)pgd, PGDIR_SHIFT,
addr, next, floor, ceiling);
}
@@ -753,12 +737,13 @@ static int __init add_huge_page_size(unsigned long long size)
/* Check that it is a page size supported by the hardware and
* that it fits within pagetable and slice limits. */
+ if (size <= PAGE_SIZE)
+ return -EINVAL;
#ifdef CONFIG_PPC_FSL_BOOK3E
- if ((size < PAGE_SIZE) || !is_power_of_4(size))
+ if (!is_power_of_4(size))
return -EINVAL;
#else
- if (!is_power_of_2(size)
- || (shift > SLICE_HIGH_SHIFT) || (shift <= PAGE_SHIFT))
+ if (!is_power_of_2(size) || (shift > SLICE_HIGH_SHIFT))
return -EINVAL;
#endif
@@ -791,53 +776,15 @@ static int __init hugepage_setup_sz(char *str)
}
__setup("hugepagesz=", hugepage_setup_sz);
-#ifdef CONFIG_PPC_FSL_BOOK3E
struct kmem_cache *hugepte_cache;
static int __init hugetlbpage_init(void)
{
int psize;
- for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
- unsigned shift;
-
- if (!mmu_psize_defs[psize].shift)
- continue;
-
- shift = mmu_psize_to_shift(psize);
-
- /* Don't treat normal page sizes as huge... */
- if (shift != PAGE_SHIFT)
- if (add_huge_page_size(1ULL << shift) < 0)
- continue;
- }
-
- /*
- * Create a kmem cache for hugeptes. The bottom bits in the pte have
- * size information encoded in them, so align them to allow this
- */
- hugepte_cache = kmem_cache_create("hugepte-cache", sizeof(pte_t),
- HUGEPD_SHIFT_MASK + 1, 0, NULL);
- if (hugepte_cache == NULL)
- panic("%s: Unable to create kmem cache for hugeptes\n",
- __func__);
-
- /* Default hpage size = 4M */
- if (mmu_psize_defs[MMU_PAGE_4M].shift)
- HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_4M].shift;
- else
- panic("%s: Unable to set default huge page size\n", __func__);
-
-
- return 0;
-}
-#else
-static int __init hugetlbpage_init(void)
-{
- int psize;
-
+#if !defined(CONFIG_PPC_FSL_BOOK3E)
if (!radix_enabled() && !mmu_has_feature(MMU_FTR_16M_PAGE))
return -ENODEV;
-
+#endif
for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
unsigned shift;
unsigned pdshift;
@@ -860,16 +807,31 @@ static int __init hugetlbpage_init(void)
* if we have pdshift and shift value same, we don't
* use pgt cache for hugepd.
*/
- if (pdshift != shift) {
+ if (pdshift > shift) {
pgtable_cache_add(pdshift - shift, NULL);
if (!PGT_CACHE(pdshift - shift))
panic("hugetlbpage_init(): could not create "
"pgtable cache for %d bit pagesize\n", shift);
+ } else if (!hugepte_cache) {
+ /*
+ * Create a kmem cache for hugeptes. The bottom bits in
+ * the pte have size information encoded in them, so
+ * align them to allow this
+ */
+ hugepte_cache = kmem_cache_create("hugepte-cache",
+ sizeof(pte_t),
+ HUGEPD_SHIFT_MASK + 1,
+ 0, NULL);
+ if (hugepte_cache == NULL)
+ panic("%s: Unable to create kmem cache "
+ "for hugeptes\n", __func__);
+
}
}
/* Set default large page size. Currently, we pick 16M or 1M
* depending on what is available
+ * We select 4M on other ones.
*/
if (mmu_psize_defs[MMU_PAGE_16M].shift)
HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_16M].shift;
@@ -877,11 +839,14 @@ static int __init hugetlbpage_init(void)
HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_1M].shift;
else if (mmu_psize_defs[MMU_PAGE_2M].shift)
HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_2M].shift;
-
+ else if (mmu_psize_defs[MMU_PAGE_4M].shift)
+ HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_4M].shift;
+ else
+ panic("%s: Unable to set default huge page size\n", __func__);
return 0;
}
-#endif
+
arch_initcall(hugetlbpage_init);
void flush_dcache_icache_hugepage(struct page *page)
--
2.1.0
[toc] | [prev] | [next] | [standalone]
| From | "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> |
|---|---|
| Date | 2016-09-19 07:50 +0200 |
| Subject | Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic |
| Message-ID | <siZCN-1pF-1@gated-at.bofh.it> |
| In reply to | #1484748 |
Christophe Leroy <christophe.leroy@c-s.fr> writes:
> Today there are two implementations of hugetlbpages which are managed
> by exclusive #ifdefs:
> * FSL_BOOKE: several directory entries points to the same single hugepage
> * BOOK3S: one upper level directory entry points to a table of hugepages
>
> In preparation of implementation of hugepage support on the 8xx, we
> need a mix of the two above solutions, because the 8xx needs both cases
> depending on the size of pages:
> * In 4k page size mode, each PGD entry covers a 4M bytes area. It means
> that 2 PGD entries will be necessary to cover an 8M hugepage while a
> single PGD entry will cover 8x 512k hugepages.
> * In 16 page size mode, each PGD entry covers a 64M bytes area. It means
> that 8x 8M hugepages will be covered by one PGD entry and 64x 512k
> hugepages will be covers by one PGD entry.
>
> This patch:
> * removes #ifdefs in favor of if/else based on the range sizes
> * merges the two huge_pte_alloc() functions as they are pretty similar
> * merges the two hugetlbpage_init() functions as they are pretty similar
>
> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
> ---
> v2: This part is new and results from a split of last patch of v1 serie in
> two parts
>
> arch/powerpc/mm/hugetlbpage.c | 189 +++++++++++++++++-------------------------
> 1 file changed, 77 insertions(+), 112 deletions(-)
>
> diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
> index 8a512b1..2119f00 100644
> --- a/arch/powerpc/mm/hugetlbpage.c
> +++ b/arch/powerpc/mm/hugetlbpage.c
> @@ -64,14 +64,16 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
> {
> struct kmem_cache *cachep;
> pte_t *new;
> -
> -#ifdef CONFIG_PPC_FSL_BOOK3E
> int i;
> - int num_hugepd = 1 << (pshift - pdshift);
> - cachep = hugepte_cache;
> -#else
> - cachep = PGT_CACHE(pdshift - pshift);
> -#endif
> + int num_hugepd;
> +
> + if (pshift >= pdshift) {
> + cachep = hugepte_cache;
> + num_hugepd = 1 << (pshift - pdshift);
> + } else {
> + cachep = PGT_CACHE(pdshift - pshift);
> + num_hugepd = 1;
> + }
Is there a way to hint likely/unlikely branch based on the page size
selected at build time ?
>
> new = kmem_cache_zalloc(cachep, GFP_KERNEL);
>
> @@ -89,7 +91,7 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
> smp_wmb();
>
> spin_lock(&mm->page_table_lock);
> -#ifdef CONFIG_PPC_FSL_BOOK3E
> +
> /*
> * We have multiple higher-level entries that point to the same
> * actual pte location. Fill in each as we go and backtrack on error.
> @@ -100,8 +102,13 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
> if (unlikely(!hugepd_none(*hpdp)))
> break;
> else
....
> -#ifdef CONFIG_PPC_FSL_BOOK3E
> struct kmem_cache *hugepte_cache;
> static int __init hugetlbpage_init(void)
> {
> int psize;
>
> - for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
> - unsigned shift;
> -
> - if (!mmu_psize_defs[psize].shift)
> - continue;
> -
> - shift = mmu_psize_to_shift(psize);
> -
> - /* Don't treat normal page sizes as huge... */
> - if (shift != PAGE_SHIFT)
> - if (add_huge_page_size(1ULL << shift) < 0)
> - continue;
> - }
> -
> - /*
> - * Create a kmem cache for hugeptes. The bottom bits in the pte have
> - * size information encoded in them, so align them to allow this
> - */
> - hugepte_cache = kmem_cache_create("hugepte-cache", sizeof(pte_t),
> - HUGEPD_SHIFT_MASK + 1, 0, NULL);
> - if (hugepte_cache == NULL)
> - panic("%s: Unable to create kmem cache for hugeptes\n",
> - __func__);
> -
> - /* Default hpage size = 4M */
> - if (mmu_psize_defs[MMU_PAGE_4M].shift)
> - HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_4M].shift;
> - else
> - panic("%s: Unable to set default huge page size\n", __func__);
> -
> -
> - return 0;
> -}
> -#else
> -static int __init hugetlbpage_init(void)
> -{
> - int psize;
> -
> +#if !defined(CONFIG_PPC_FSL_BOOK3E)
> if (!radix_enabled() && !mmu_has_feature(MMU_FTR_16M_PAGE))
> return -ENODEV;
> -
> +#endif
Do we need that #if ? radix_enabled() should become 0 and that if
condition should be removed at compile time isn't it ? or are you
finding errors with that ?
> for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
> unsigned shift;
> unsigned pdshift;
> @@ -860,16 +807,31 @@ static int __init hugetlbpage_init(void)
> * if we have pdshift and shift value same, we don't
> * use pgt cache for hugepd.
> */
> - if (pdshift != shift) {
> + if (pdshift > shift) {
> pgtable_cache_add(pdshift - shift, NULL);
> if (!PGT_CACHE(pdshift - shift))
> panic("hugetlbpage_init(): could not create "
> "pgtable cache for %d bit pagesize\n", shift);
> + } else if (!hugepte_cache) {
> + /*
> + * Create a kmem cache for hugeptes. The bottom bits in
> + * the pte have size information encoded in them, so
> + * align them to allow this
> + */
> + hugepte_cache = kmem_cache_create("hugepte-cache",
> + sizeof(pte_t),
> + HUGEPD_SHIFT_MASK + 1,
> + 0, NULL);
> + if (hugepte_cache == NULL)
> + panic("%s: Unable to create kmem cache "
> + "for hugeptes\n", __func__);
> +
We don't need hugepte_cache for book3s 64K. I guess we will endup
creating one here ?
> }
> }
>
> /* Set default large page size. Currently, we pick 16M or 1M
> * depending on what is available
> + * We select 4M on other ones.
> */
> if (mmu_psize_defs[MMU_PAGE_16M].shift)
> HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_16M].shift;
> @@ -877,11 +839,14 @@ static int __init hugetlbpage_init(void)
> HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_1M].shift;
> else if (mmu_psize_defs[MMU_PAGE_2M].shift)
> HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_2M].shift;
> -
> + else if (mmu_psize_defs[MMU_PAGE_4M].shift)
> + HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_4M].shift;
> + else
> + panic("%s: Unable to set default huge page size\n", __func__);
>
> return 0;
> }
> -#endif
> +
> arch_initcall(hugetlbpage_init);
>
> void flush_dcache_icache_hugepage(struct page *page)
> --
> 2.1.0
[toc] | [prev] | [next] | [standalone]
| From | christophe leroy <christophe.leroy@c-s.fr> |
|---|---|
| Date | 2016-09-19 20:40 +0200 |
| Subject | Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic |
| Message-ID | <sjbDY-y8-35@gated-at.bofh.it> |
| In reply to | #1486149 |
Le 19/09/2016 à 07:45, Aneesh Kumar K.V a écrit :
> Christophe Leroy <christophe.leroy@c-s.fr> writes:
>
>> Today there are two implementations of hugetlbpages which are managed
>> by exclusive #ifdefs:
>> * FSL_BOOKE: several directory entries points to the same single hugepage
>> * BOOK3S: one upper level directory entry points to a table of hugepages
>>
>> In preparation of implementation of hugepage support on the 8xx, we
>> need a mix of the two above solutions, because the 8xx needs both cases
>> depending on the size of pages:
>> * In 4k page size mode, each PGD entry covers a 4M bytes area. It means
>> that 2 PGD entries will be necessary to cover an 8M hugepage while a
>> single PGD entry will cover 8x 512k hugepages.
>> * In 16 page size mode, each PGD entry covers a 64M bytes area. It means
>> that 8x 8M hugepages will be covered by one PGD entry and 64x 512k
>> hugepages will be covers by one PGD entry.
>>
>> This patch:
>> * removes #ifdefs in favor of if/else based on the range sizes
>> * merges the two huge_pte_alloc() functions as they are pretty similar
>> * merges the two hugetlbpage_init() functions as they are pretty similar
>>
>> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
>> ---
>> v2: This part is new and results from a split of last patch of v1 serie in
>> two parts
>>
>> arch/powerpc/mm/hugetlbpage.c | 189 +++++++++++++++++-------------------------
>> 1 file changed, 77 insertions(+), 112 deletions(-)
>>
>> diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
>> index 8a512b1..2119f00 100644
>> --- a/arch/powerpc/mm/hugetlbpage.c
>> +++ b/arch/powerpc/mm/hugetlbpage.c
[...]
>
>> -#ifdef CONFIG_PPC_FSL_BOOK3E
>> struct kmem_cache *hugepte_cache;
>> static int __init hugetlbpage_init(void)
>> {
>> int psize;
>>
>> - for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
>> - unsigned shift;
>> -
>> - if (!mmu_psize_defs[psize].shift)
>> - continue;
>> -
>> - shift = mmu_psize_to_shift(psize);
>> -
>> - /* Don't treat normal page sizes as huge... */
>> - if (shift != PAGE_SHIFT)
>> - if (add_huge_page_size(1ULL << shift) < 0)
>> - continue;
>> - }
>> -
>> - /*
>> - * Create a kmem cache for hugeptes. The bottom bits in the pte have
>> - * size information encoded in them, so align them to allow this
>> - */
>> - hugepte_cache = kmem_cache_create("hugepte-cache", sizeof(pte_t),
>> - HUGEPD_SHIFT_MASK + 1, 0, NULL);
>> - if (hugepte_cache == NULL)
>> - panic("%s: Unable to create kmem cache for hugeptes\n",
>> - __func__);
>> -
>> - /* Default hpage size = 4M */
>> - if (mmu_psize_defs[MMU_PAGE_4M].shift)
>> - HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_4M].shift;
>> - else
>> - panic("%s: Unable to set default huge page size\n", __func__);
>> -
>> -
>> - return 0;
>> -}
>> -#else
>> -static int __init hugetlbpage_init(void)
>> -{
>> - int psize;
>> -
>> +#if !defined(CONFIG_PPC_FSL_BOOK3E)
>> if (!radix_enabled() && !mmu_has_feature(MMU_FTR_16M_PAGE))
>> return -ENODEV;
>> -
>> +#endif
>
> Do we need that #if ? radix_enabled() should become 0 and that if
> condition should be removed at compile time isn't it ? or are you
> finding errors with that ?
Having radix_enabled() being 0, it becomes:
if (!mmu_has_feature(MMU_FTR_16M_PAGE))
return -ENODEV;
Which means hugepage will only be handled by CPUs having 16M pages.
That's the issue.
>
>
>> for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
>> unsigned shift;
>> unsigned pdshift;
>> @@ -860,16 +807,31 @@ static int __init hugetlbpage_init(void)
>> * if we have pdshift and shift value same, we don't
>> * use pgt cache for hugepd.
>> */
>> - if (pdshift != shift) {
>> + if (pdshift > shift) {
>> pgtable_cache_add(pdshift - shift, NULL);
>> if (!PGT_CACHE(pdshift - shift))
>> panic("hugetlbpage_init(): could not create "
>> "pgtable cache for %d bit pagesize\n", shift);
>> + } else if (!hugepte_cache) {
>> + /*
>> + * Create a kmem cache for hugeptes. The bottom bits in
>> + * the pte have size information encoded in them, so
>> + * align them to allow this
>> + */
>> + hugepte_cache = kmem_cache_create("hugepte-cache",
>> + sizeof(pte_t),
>> + HUGEPD_SHIFT_MASK + 1,
>> + 0, NULL);
>> + if (hugepte_cache == NULL)
>> + panic("%s: Unable to create kmem cache "
>> + "for hugeptes\n", __func__);
>> +
>
>
> We don't need hugepte_cache for book3s 64K. I guess we will endup
> creating one here ?
Should not, because on book3s 64k, we will have pdshift > shift
won't we ?
>
>> }
>> }
>>
>> /* Set default large page size. Currently, we pick 16M or 1M
>> * depending on what is available
>> + * We select 4M on other ones.
>> */
>> if (mmu_psize_defs[MMU_PAGE_16M].shift)
>> HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_16M].shift;
>> @@ -877,11 +839,14 @@ static int __init hugetlbpage_init(void)
>> HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_1M].shift;
>> else if (mmu_psize_defs[MMU_PAGE_2M].shift)
>> HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_2M].shift;
>> -
>> + else if (mmu_psize_defs[MMU_PAGE_4M].shift)
>> + HPAGE_SHIFT = mmu_psize_defs[MMU_PAGE_4M].shift;
>> + else
>> + panic("%s: Unable to set default huge page size\n", __func__);
>>
>> return 0;
>> }
>> -#endif
>> +
>> arch_initcall(hugetlbpage_init);
>>
>> void flush_dcache_icache_hugepage(struct page *page)
>> --
>> 2.1.0
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
[toc] | [prev] | [next] | [standalone]
| From | "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> |
|---|---|
| Date | 2016-09-20 04:50 +0200 |
| Subject | Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic |
| Message-ID | <sjjia-5z5-3@gated-at.bofh.it> |
| In reply to | #1486757 |
christophe leroy <christophe.leroy@c-s.fr> writes:
>>
>>
>>> for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
>>> unsigned shift;
>>> unsigned pdshift;
>>> @@ -860,16 +807,31 @@ static int __init hugetlbpage_init(void)
>>> * if we have pdshift and shift value same, we don't
>>> * use pgt cache for hugepd.
>>> */
>>> - if (pdshift != shift) {
>>> + if (pdshift > shift) {
>>> pgtable_cache_add(pdshift - shift, NULL);
>>> if (!PGT_CACHE(pdshift - shift))
>>> panic("hugetlbpage_init(): could not create "
>>> "pgtable cache for %d bit pagesize\n", shift);
>>> + } else if (!hugepte_cache) {
>>> + /*
>>> + * Create a kmem cache for hugeptes. The bottom bits in
>>> + * the pte have size information encoded in them, so
>>> + * align them to allow this
>>> + */
>>> + hugepte_cache = kmem_cache_create("hugepte-cache",
>>> + sizeof(pte_t),
>>> + HUGEPD_SHIFT_MASK + 1,
>>> + 0, NULL);
>>> + if (hugepte_cache == NULL)
>>> + panic("%s: Unable to create kmem cache "
>>> + "for hugeptes\n", __func__);
>>> +
>>
>>
>> We don't need hugepte_cache for book3s 64K. I guess we will endup
>> creating one here ?
>
> Should not, because on book3s 64k, we will have pdshift > shift
> won't we ?
>
on 64k book3s, we have pdshift == shift and we don't need to create
hugepd cache on book3s 64k.
-aneesh
[toc] | [prev] | [next] | [standalone]
| From | Christophe Leroy <christophe.leroy@c-s.fr> |
|---|---|
| Date | 2016-09-21 08:20 +0200 |
| Subject | Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic |
| Message-ID | <sjJ2W-5pe-3@gated-at.bofh.it> |
| In reply to | #1486149 |
Le 19/09/2016 à 07:45, Aneesh Kumar K.V a écrit :
> Christophe Leroy <christophe.leroy@c-s.fr> writes:
>
>> Today there are two implementations of hugetlbpages which are managed
>> by exclusive #ifdefs:
>> * FSL_BOOKE: several directory entries points to the same single hugepage
>> * BOOK3S: one upper level directory entry points to a table of hugepages
>>
>> In preparation of implementation of hugepage support on the 8xx, we
>> need a mix of the two above solutions, because the 8xx needs both cases
>> depending on the size of pages:
>> * In 4k page size mode, each PGD entry covers a 4M bytes area. It means
>> that 2 PGD entries will be necessary to cover an 8M hugepage while a
>> single PGD entry will cover 8x 512k hugepages.
>> * In 16 page size mode, each PGD entry covers a 64M bytes area. It means
>> that 8x 8M hugepages will be covered by one PGD entry and 64x 512k
>> hugepages will be covers by one PGD entry.
>>
>> This patch:
>> * removes #ifdefs in favor of if/else based on the range sizes
>> * merges the two huge_pte_alloc() functions as they are pretty similar
>> * merges the two hugetlbpage_init() functions as they are pretty similar
>>
>> Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
>> ---
>> v2: This part is new and results from a split of last patch of v1 serie in
>> two parts
>>
>> arch/powerpc/mm/hugetlbpage.c | 189 +++++++++++++++++-------------------------
>> 1 file changed, 77 insertions(+), 112 deletions(-)
>>
>> diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c
>> index 8a512b1..2119f00 100644
>> --- a/arch/powerpc/mm/hugetlbpage.c
>> +++ b/arch/powerpc/mm/hugetlbpage.c
>> @@ -64,14 +64,16 @@ static int __hugepte_alloc(struct mm_struct *mm, hugepd_t *hpdp,
>> {
>> struct kmem_cache *cachep;
>> pte_t *new;
>> -
>> -#ifdef CONFIG_PPC_FSL_BOOK3E
>> int i;
>> - int num_hugepd = 1 << (pshift - pdshift);
>> - cachep = hugepte_cache;
>> -#else
>> - cachep = PGT_CACHE(pdshift - pshift);
>> -#endif
>> + int num_hugepd;
>> +
>> + if (pshift >= pdshift) {
>> + cachep = hugepte_cache;
>> + num_hugepd = 1 << (pshift - pdshift);
>> + } else {
>> + cachep = PGT_CACHE(pdshift - pshift);
>> + num_hugepd = 1;
>> + }
>
> Is there a way to hint likely/unlikely branch based on the page size
> selected at build time ?
Is that really worth it, won't it be negligeable compared to other
actions in that function (like for instance kmem_cache_zalloc()) ?
Can't we just trust GCC on that one ?
Christophe
[toc] | [prev] | [next] | [standalone]
| From | "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> |
|---|---|
| Date | 2016-09-19 08:00 +0200 |
| Subject | Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic |
| Message-ID | <siZMu-1sT-7@gated-at.bofh.it> |
| In reply to | #1484748 |
Christophe Leroy <christophe.leroy@c-s.fr> writes:
> +#else
> +static void hugepd_free(struct mmu_gather *tlb, void *hugepte)
> +{
> + BUG();
> +}
> +
> #endif
I was expecting that BUG will get removed in the next patch. But I don't
see it in the next patch. Considering
@@ -475,11 +453,10 @@ static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshif
for (i = 0; i < num_hugepd; i++, hpdp++)
hpdp->pd = 0;
-#ifdef CONFIG_PPC_FSL_BOOK3E
- hugepd_free(tlb, hugepte);
-#else
- pgtable_free_tlb(tlb, hugepte, pdshift - shift);
-#endif
+ if (shift >= pdshift)
+ hugepd_free(tlb, hugepte);
+ else
+ pgtable_free_tlb(tlb, hugepte, pdshift - shift);
}
What is that I am missing ?
-aneesh
[toc] | [prev] | [next] | [standalone]
| From | christophe leroy <christophe.leroy@c-s.fr> |
|---|---|
| Date | 2016-09-19 20:40 +0200 |
| Subject | Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic |
| Message-ID | <sjbDY-y8-33@gated-at.bofh.it> |
| In reply to | #1486152 |
Le 19/09/2016 à 07:50, Aneesh Kumar K.V a écrit :
>
> Christophe Leroy <christophe.leroy@c-s.fr> writes:
>> +#else
>> +static void hugepd_free(struct mmu_gather *tlb, void *hugepte)
>> +{
>> + BUG();
>> +}
>> +
>> #endif
>
>
> I was expecting that BUG will get removed in the next patch. But I don't
> see it in the next patch. Considering
>
> @@ -475,11 +453,10 @@ static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshif
> for (i = 0; i < num_hugepd; i++, hpdp++)
> hpdp->pd = 0;
>
> -#ifdef CONFIG_PPC_FSL_BOOK3E
> - hugepd_free(tlb, hugepte);
> -#else
> - pgtable_free_tlb(tlb, hugepte, pdshift - shift);
> -#endif
> + if (shift >= pdshift)
> + hugepd_free(tlb, hugepte);
> + else
> + pgtable_free_tlb(tlb, hugepte, pdshift - shift);
> }
>
> What is that I am missing ?
>
Previously, call to hugepd_free() was compiled only when #ifdef
CONFIG_PPC_FSL_BOOK3E
Now, it is compiled at all time, but it should never be called if not
CONFIG_PPC_FSL_BOOK3E because we always have shift < pdshift in that case.
Then the function needs to be defined anyway but should never be called.
Should I just define it static inline {} ?
Christophe
---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
[toc] | [prev] | [next] | [standalone]
| From | "Aneesh Kumar K.V" <aneesh.kumar@linux.vnet.ibm.com> |
|---|---|
| Date | 2016-09-20 04:30 +0200 |
| Subject | Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic |
| Message-ID | <sjiYN-5sN-7@gated-at.bofh.it> |
| In reply to | #1486756 |
christophe leroy <christophe.leroy@c-s.fr> writes:
> Le 19/09/2016 à 07:50, Aneesh Kumar K.V a écrit :
>>
>> Christophe Leroy <christophe.leroy@c-s.fr> writes:
>>> +#else
>>> +static void hugepd_free(struct mmu_gather *tlb, void *hugepte)
>>> +{
>>> + BUG();
>>> +}
>>> +
>>> #endif
>>
>>
>> I was expecting that BUG will get removed in the next patch. But I don't
>> see it in the next patch. Considering
>>
>> @@ -475,11 +453,10 @@ static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshif
>> for (i = 0; i < num_hugepd; i++, hpdp++)
>> hpdp->pd = 0;
>>
>> -#ifdef CONFIG_PPC_FSL_BOOK3E
>> - hugepd_free(tlb, hugepte);
>> -#else
>> - pgtable_free_tlb(tlb, hugepte, pdshift - shift);
>> -#endif
>> + if (shift >= pdshift)
>> + hugepd_free(tlb, hugepte);
>> + else
>> + pgtable_free_tlb(tlb, hugepte, pdshift - shift);
>> }
>>
>> What is that I am missing ?
>>
>
> Previously, call to hugepd_free() was compiled only when #ifdef
> CONFIG_PPC_FSL_BOOK3E
> Now, it is compiled at all time, but it should never be called if not
> CONFIG_PPC_FSL_BOOK3E because we always have shift < pdshift in that case.
> Then the function needs to be defined anyway but should never be called.
> Should I just define it static inline {} ?
>
For 8M with 4K mode, we have shift >= pdshift right ?
-aneesh
[toc] | [prev] | [next] | [standalone]
| From | Christophe Leroy <christophe.leroy@c-s.fr> |
|---|---|
| Date | 2016-09-20 07:30 +0200 |
| Subject | Re: [PATCH v2 2/3] powerpc: get hugetlbpage handling more generic |
| Message-ID | <sjlMZ-7eb-9@gated-at.bofh.it> |
| In reply to | #1487017 |
Le 20/09/2016 à 04:28, Aneesh Kumar K.V a écrit :
> christophe leroy <christophe.leroy@c-s.fr> writes:
>
>> Le 19/09/2016 à 07:50, Aneesh Kumar K.V a écrit :
>>>
>>> Christophe Leroy <christophe.leroy@c-s.fr> writes:
>>>> +#else
>>>> +static void hugepd_free(struct mmu_gather *tlb, void *hugepte)
>>>> +{
>>>> + BUG();
>>>> +}
>>>> +
>>>> #endif
>>>
>>>
>>> I was expecting that BUG will get removed in the next patch. But I don't
>>> see it in the next patch. Considering
>>>
>>> @@ -475,11 +453,10 @@ static void free_hugepd_range(struct mmu_gather *tlb, hugepd_t *hpdp, int pdshif
>>> for (i = 0; i < num_hugepd; i++, hpdp++)
>>> hpdp->pd = 0;
>>>
>>> -#ifdef CONFIG_PPC_FSL_BOOK3E
>>> - hugepd_free(tlb, hugepte);
>>> -#else
>>> - pgtable_free_tlb(tlb, hugepte, pdshift - shift);
>>> -#endif
>>> + if (shift >= pdshift)
>>> + hugepd_free(tlb, hugepte);
>>> + else
>>> + pgtable_free_tlb(tlb, hugepte, pdshift - shift);
>>> }
>>>
>>> What is that I am missing ?
>>>
>>
>> Previously, call to hugepd_free() was compiled only when #ifdef
>> CONFIG_PPC_FSL_BOOK3E
>> Now, it is compiled at all time, but it should never be called if not
>> CONFIG_PPC_FSL_BOOK3E because we always have shift < pdshift in that case.
>> Then the function needs to be defined anyway but should never be called.
>> Should I just define it static inline {} ?
>>
>
> For 8M with 4K mode, we have shift >= pdshift right ?
>
Yes, thats the reason why in the following patch we get. That way we get
a real hugepd_free() also for the 8xx.
@@ -366,7 +373,7 @@ int alloc_bootmem_huge_page(struct hstate *hstate)
}
#endif
-#ifdef CONFIG_PPC_FSL_BOOK3E
+#if defined(CONFIG_PPC_FSL_BOOK3E) || defined(CONFIG_PPC_8xx)
#define HUGEPD_FREELIST_SIZE \
((PAGE_SIZE - sizeof(struct hugepd_freelist)) / sizeof(pte_t))
Christophe
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web