Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1600165

[tip:x86/mm] x86/mm/gup: Add 5-level paging support

From "tip-bot for Kirill A. Shutemov" <tipbot@zytor.com>
Newsgroups linux.kernel
Subject [tip:x86/mm] x86/mm/gup: Add 5-level paging support
Date 2017-03-14 10:50 +0100
Message-ID <tkRfA-6BP-21@gated-at.bofh.it> (permalink)
References <tkziH-1PZ-41@gated-at.bofh.it>
Organization linux.* mail to news gateway

Show all headers | View raw

Commit-ID:  0318e5abe1c0933b8bf6763a1a0d3caec4f0826d
Gitweb:     http://git.kernel.org/tip/0318e5abe1c0933b8bf6763a1a0d3caec4f0826d
Author:     Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
AuthorDate: Mon, 13 Mar 2017 17:33:06 +0300
Committer:  Ingo Molnar <mingo@kernel.org>
CommitDate: Tue, 14 Mar 2017 08:45:08 +0100

x86/mm/gup: Add 5-level paging support

Extend get_user_pages_fast() to handle an additional page table level.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andy Lutomirski <luto@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Brian Gerst <brgerst@gmail.com>
Cc: Dave Hansen <dave.hansen@intel.com>
Cc: Denys Vlasenko <dvlasenk@redhat.com>
Cc: H. Peter Anvin <hpa@zytor.com>
Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-arch@vger.kernel.org
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/20170313143309.16020-4-kirill.shutemov@linux.intel.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
 arch/x86/mm/gup.c | 33 +++++++++++++++++++++++++++------
 1 file changed, 27 insertions(+), 6 deletions(-)

diff --git a/arch/x86/mm/gup.c b/arch/x86/mm/gup.c
index 1f3b6ef..456dfdf 100644
--- a/arch/x86/mm/gup.c
+++ b/arch/x86/mm/gup.c
@@ -76,9 +76,9 @@ static void undo_dev_pagemap(int *nr, int nr_start, struct page **pages)
 }
 
 /*
- * 'pteval' can come from a pte, pmd or pud.  We only check
+ * 'pteval' can come from a pte, pmd, pud or p4d.  We only check
  * _PAGE_PRESENT, _PAGE_USER, and _PAGE_RW in here which are the
- * same value on all 3 types.
+ * same value on all 4 types.
  */
 static inline int pte_allows_gup(unsigned long pteval, int write)
 {
@@ -295,13 +295,13 @@ static noinline int gup_huge_pud(pud_t pud, unsigned long addr,
 	return 1;
 }
 
-static int gup_pud_range(pgd_t pgd, unsigned long addr, unsigned long end,
+static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end,
 			int write, struct page **pages, int *nr)
 {
 	unsigned long next;
 	pud_t *pudp;
 
-	pudp = pud_offset(&pgd, addr);
+	pudp = pud_offset(&p4d, addr);
 	do {
 		pud_t pud = *pudp;
 
@@ -320,6 +320,27 @@ static int gup_pud_range(pgd_t pgd, unsigned long addr, unsigned long end,
 	return 1;
 }
 
+static int gup_p4d_range(pgd_t pgd, unsigned long addr, unsigned long end,
+			int write, struct page **pages, int *nr)
+{
+	unsigned long next;
+	p4d_t *p4dp;
+
+	p4dp = p4d_offset(&pgd, addr);
+	do {
+		p4d_t p4d = *p4dp;
+
+		next = p4d_addr_end(addr, end);
+		if (p4d_none(p4d))
+			return 0;
+		BUILD_BUG_ON(p4d_large(p4d));
+		if (!gup_pud_range(p4d, addr, next, write, pages, nr))
+			return 0;
+	} while (p4dp++, addr = next, addr != end);
+
+	return 1;
+}
+
 /*
  * Like get_user_pages_fast() except its IRQ-safe in that it won't fall
  * back to the regular GUP.
@@ -368,7 +389,7 @@ int __get_user_pages_fast(unsigned long start, int nr_pages, int write,
 		next = pgd_addr_end(addr, end);
 		if (pgd_none(pgd))
 			break;
-		if (!gup_pud_range(pgd, addr, next, write, pages, &nr))
+		if (!gup_p4d_range(pgd, addr, next, write, pages, &nr))
 			break;
 	} while (pgdp++, addr = next, addr != end);
 	local_irq_restore(flags);
@@ -440,7 +461,7 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int write,
 		next = pgd_addr_end(addr, end);
 		if (pgd_none(pgd))
 			goto slow;
-		if (!gup_pud_range(pgd, addr, next, write, pages, &nr))
+		if (!gup_p4d_range(pgd, addr, next, write, pages, &nr))
 			goto slow;
 	} while (pgdp++, addr = next, addr != end);
 	local_irq_enable();

Back to linux.kernel | Previous | Next — Previous in thread | Next in thread | Find similar | Unroll thread

Thread

[PATCH 0/6] x86: 5-level paging enabling for v4.12, Part 1 "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2017-03-13 15:40 +0100
  [PATCH 4/6] x86/ident_map: Add 5-level paging support "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2017-03-13 15:40 +0100
    [tip:x86/mm] x86/mm/ident_map: Add 5-level paging support "tip-bot for Kirill A. Shutemov" <tipbot@zytor.com> - 2017-03-14 10:50 +0100
  [PATCH 1/6] x86/mm: Extend headers with basic definitions to support 5-level paging "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2017-03-13 15:40 +0100
    [tip:x86/mm] x86/mm: Extend headers with basic definitions to  support 5-level paging "tip-bot for Kirill A. Shutemov" <tipbot@zytor.com> - 2017-03-14 10:50 +0100
  [PATCH 5/6] x86/vmalloc: Add 5-level paging support "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2017-03-13 15:40 +0100
    [tip:x86/mm] x86/mm/vmalloc: Add 5-level paging support "tip-bot for Kirill A. Shutemov" <tipbot@zytor.com> - 2017-03-14 10:50 +0100
  [PATCH 3/6] x86/gup: Add 5-level paging support "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2017-03-13 15:40 +0100
    [tip:x86/mm] x86/mm/gup: Add 5-level paging support "tip-bot for Kirill A. Shutemov" <tipbot@zytor.com> - 2017-03-14 10:50 +0100
  [PATCH 6/6] x86/power: Add 5-level paging support "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2017-03-13 15:40 +0100
    [tip:x86/mm] x86/power: Add 5-level paging support "tip-bot for Kirill A. Shutemov" <tipbot@zytor.com> - 2017-03-14 10:50 +0100
  [PATCH 2/6] x86/mm: Convert trivial cases of page table walk to 5-level paging "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com> - 2017-03-13 15:40 +0100
    [tip:x86/mm] x86/mm: Convert trivial cases of page table walk to  5-level paging "tip-bot for Kirill A. Shutemov" <tipbot@zytor.com> - 2017-03-14 10:40 +0100
  Re: [PATCH 0/6] x86: 5-level paging enabling for v4.12, Part 1 Linus Torvalds <torvalds@linux-foundation.org> - 2017-03-13 20:50 +0100
  Re: [PATCH 0/6] x86: 5-level paging enabling for v4.12, Part 1 Ingo Molnar <mingo@kernel.org> - 2017-03-14 08:50 +0100
    Re: [PATCH 0/6] x86: 5-level paging enabling for v4.12, Part 1 "Kirill A. Shutemov" <kirill@shutemov.name> - 2017-03-14 09:30 +0100
      Re: [PATCH 0/6] x86: 5-level paging enabling for v4.12, Part 1 Thomas Gleixner <tglx@linutronix.de> - 2017-03-14 09:40 +0100
    Re: [PATCH 0/6] x86: 5-level paging enabling for v4.12, Part 1 Linus Torvalds <torvalds@linux-foundation.org> - 2017-03-14 18:50 +0100
      Re: [PATCH 0/6] x86: 5-level paging enabling for v4.12, Part 1 "Kirill A. Shutemov" <kirill@shutemov.name> - 2017-03-18 18:10 +0100
      Re: [PATCH 0/6] x86: 5-level paging enabling for v4.12, Part 1 "Kirill A. Shutemov" <kirill@shutemov.name> - 2017-03-18 18:10 +0100
    Re: [PATCH 0/6] x86: 5-level paging enabling for v4.12, Part 1 Michal Hocko <mhocko@kernel.org> - 2017-03-15 10:30 +0100

csiph-web